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BEST  PRACTICES 

DOD  Can  Achieve  Better  Outcomes  by  Standardizing 
the  Way  Manufacturing  Risks  Are  Managed 


Why  GAO  Did  This  Study 

Cost  growth  and  schedule  delays 
are  prevalent  problems  in  acquiring 
defense  weapon  systems. 
Manufacturing  systems  has  proven 
difficult,  particularly  as  programs 
transition  to  production.  In 
December  2008,  the  Department  of 
Defense  (DOD)  issued  an  updated 
version  of  its  acquisition  policy  that 
reflects  earlier  consideration  of 
manufacturing  risks.  A  joint 
defense  and  industry  group 
developed  manufacturing  readiness 
levels  (MRL)  to  support 
assessments  of  manufacturing 
risks.  Use  of  MRLs  on  all  weapon 
acquisition  programs  has  been 
proposed.  In  response  to  a 
congressional  request,  this  report 
assesses  the  manufacturing 
problems  faced  by  DOD,  how  MRLs 
can  address  manufacturing 
problems,  how  MRLs  compare  to 
manufacturing  best  practices  of 
leading  commercial  firms,  and 
challenges  and  barriers  to 
implementing  MRLs  at  DOD.  In 
conducting  our  work,  we  contacted 
DOD,  military  services,  and 
contractors;  held  interviews  with 
leading  commercial  firms;  reviewed 
program  documents  and  policy 
proposals;  and  spoke  with 
manufacturing  experts. 


What  GAO  Recommends 


GAO  recommends  that  the 
Secretary  of  Defense  require  the 
use  of  MRLs  across  DOD  programs, 
strengthen  the  MRL  criteria 
(process  control)  for  production 
start,  assess  the  need  for  tools,  and 
assess  the  manufacturing 
workforce  to  address  knowledge 
gaps.  DOD  partially  concurred  with 
the  first  recommendation,  and 
concurred  with  the  other  three. 

View  the  full  GAO-1 0-439.  or  key 
components.  For  more  information,  contact 
Michael  Sullivan  at  (202)  512-4841  or 
sullivanm@gao.gov. 


What  GAO  Found 

DOD  faces  problems  in  manufacturing  weapon  systems — systems  cost  far  more 
and  take  much  longer  to  build  than  estimated.  Billions  of  dollars  in  cost  growth 
occur  as  programs  transition  from  development  to  production,  and  unit-cost 
increases  are  common  after  production  begins.  Several  factors  contribute  to  these 
problems  including  inattention  to  manufacturing  during  planning  and  design,  poor 
supplier  management,  and  a  deficit  in  manufacturing  knowledge  among  the 
acquisition  workforce.  Essentially,  programs  did  not  identify  and  resolve 
manufacturing  risks  early  in  development,  but  carried  risks  into  production  where 
they  emerged  as  significant  problems. 

MRLs  have  been  proposed  as  new  criteria  for  improving  the  way  DOD  identifies 
and  manages  manufacturing  risks  and  readiness.  Introduced  to  the  defense 
community  in  2005,  MRLs  were  developed  from  an  extensive  body  of 
manufacturing  knowledge  that  includes  defense,  industry,  and  academic  sources. 
An  analysis  of  DOD’s  technical  reviews  that  assesses  how  programs  are 
progressing  show  that  MRLs  address  many  gaps  in  core  manufacturing-related 
areas,  particularly  during  the  early  acquisition  phases.  Several  Army  and  Air  Force 
centers  that  piloted  MRLs  report  these  metrics  contributed  to  substantial  cost 
benefits  on  a  variety  of  technologies  and  major  defense  acquisition  programs. 

To  develop  and  manufacture  products,  the  commercial  firms  we  visited  use  a 
disciplined,  gated  process  that  emphasizes  manufacturing  criteria  early  in 
development.  The  practices  they  employ  focus  on  gathering  sufficient  knowledge 
about  the  producibility  of  their  products  to  lower  risks,  and  include  stringent 
manufacturing  readiness  criteria  to  measure  whether  the  product  is  sufficiently 
mature  to  move  forward  in  development.  These  criteria  are  similar  to  DOD’s 
proposed  MRLs  in  that  commercial  firms 

•  assess  producibility  at  each  gate  using  clearly  defined  manufacturing 
criteria  to  gain  knowledge  about  manufacturing  early, 

•  demonstrate  manufacturing  processes  in  a  production-relevant 
environment,  and 

•  emphasize  relationships  with  critical  suppliers. 

However,  a  key  difference  is  that  commercial  firms,  prior  to  starting  production, 
require  their  manufacturing  processes  to  be  in  control — that  is,  critical  processes 
are  repeatable,  sustainable,  and  consistently  producing  parts  within  the  quality 
standards.  DOD’s  proposed  MRL  criteria  do  not  require  that  processes  be  in 
control  until  later. 

Acceptance  of  MRLs  has  grown  among  some  industry  and  DOD  components.  Yet, 
DOD  has  been  slow  to  adopt  a  policy  that  would  require  MRLs  across  DOD. 
Concerns  raised  by  the  military  services  have  centered  on  when  and  how  the  MRL 
assessments  would  be  used.  While  a  joint  DOD  and  industry  group  has  sought  to 
address  concerns  and  disseminate  information  on  benefits,  a  consensus  has  not 
been  reached.  If  adopted,  DOD  will  need  to  address  gaps  in  workforce  knowledge, 
given  the  decrease  in  the  number  of  staff  in  the  production  and  manufacturing 
career  fields. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


April  22,  2010 

The  Honorable  Bill  Nelson 
Chairman 

Subcommittee  on  Emerging  Threats  and  Capabilities 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Jack  Reed 
United  States  Senate 

The  Department  of  Defense  (DOD)  has  a  well-documented  history  of 
taking  much  longer  and  spending  much  more  than  originally  planned  to 
develop  and  acquire  its  weapons  systems.  In  particular,  as  systems 
transition  from  development  to  production,  programs  experience 
significant  manufacturing  problems.  While  DOD  has  made  some  progress 
over  the  last  two  decades  in  addressing  the  problem — including  policy 
changes  and  advocating  the  use  of  best  practices  for  product 
development — GAO’s  recent  weapon  system  reviews  show  that 
manufacturing  problems,  among  others,  continue  to  hinder  acquisition 
cost,  schedule,  and  performance  outcomes. 

It  is  essential  to  find  better  ways  of  doing  business  and,  in  particular,  to 
make  sure  systems  are  manufactured  on  time  and  cost-effectively.  To  this 
end,  leading  commercial  companies  have  achieved  more  predictable 
outcomes  from  their  manufacturing  efforts  because  they  understand 
producibility — the  relative  ease  of  producing  designs  of  an  item,  product, 
or  system  economically  with  available  production  techniques — and 
identify  manufacturing  risks  early  and  manage  them  effectively  throughout 
a  product’s  development  life  cycle. 

On  December  8,  2008,  DOD  issued  a  revised  version  of  its  policy 
instruction  on  operation  of  the  defense  acquisition  system  that,  among 
other  things,  recognizes  the  need  to  consider  manufacturing  risks  earlier 
in  the  acquisition  life  cycle  and  assesses  risks  prior  to  key  decision  points. 
In  response  to  the  need  for  the  department  to  better  design  and  produce 
more  affordable  weapon  systems,  and  to  give  decision  makers  and 
managers  better  visibility  into  their  program  risks,  a  joint  defense  and 
industry  working  group  was  established  in  2004  to  develop  manufacturing 
readiness  levels  (MRL),  a  measurement  scale  designed  to  improve  the 
management  and  communication  of  manufacturing  risk  and  readiness 
throughout  acquisitions.  Similar  to  technology  metrics  that  measure  the 
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readiness  of  a  technology,  MRLs  are  new  manufacturing  criteria  that 
measures  the  manufacturing  maturity  or  readiness  of  a  given  technology, 
manufacturing  process,  system,  or  element  of  a  weapon  system  at  various 
phases  of  the  acquisition  life  cycle. 

In  response  to  a  request  from  the  Senate  Subcommittee  on  Emerging 
Threats  and  Capabilities  and  Senator  Reed,  we  reviewed  DOD’s  efforts  to 
adopt  MRLs.  This  report  addresses  (1)  the  manufacturing  problems 
experienced  by  selected  DOD  programs,  (2)  how  MRLs  can  address  DOD’s 
manufacturing  problems,  (3)  how  proposed  MRLs  compare  to 
manufacturing  best  practices  of  leading  commercial  companies,  and  (4) 
the  challenges  and  barriers  to  implementing  MRLs  at  DOD. 

To  meet  these  objectives,  we  compared  the  manufacturing  practices  of 
DOD  and  its  large  prime  contractors  with  those  of  leading  commercial 
companies.  We  performed  an  aggregate  analysis  of  DOD  programs  from 
our  annual  weapons  assessment.1  We  also  evaluated  four  major  defense 
weapon  systems  in  production  with  known  cost  and  schedule  problems  to 
gain  in-depth  insights  as  to  the  nature  and  causes  of  problems.  We  also 
evaluated  two  defense  systems  known  to  be  producing  systems  within 
cost  and  schedule  goals  and  compared  their  practices  to  those  employed 
by  commercial  firms.  We  examined  program  documentation  and  policy 
proposals,  and  held  discussions  with  manufacturing  and  systems- 
engineering  officials  from  DOD  program  offices,  prime  contractors,  and 
the  Defense  Contract  Management  Agency.  We  also  reviewed  lessons 
learned  from  DOD  programs  that  pilot-tested  MRLs.  We  met  with  officials 
from  the  Office  of  the  Secretary  of  Defense,  Air  Force,  Army,  and  Navy, 
Missile  Defense  Agency,  Joint  Defense  Manufacturing  Technology  Panel, 
Defense  Acquisition  University,  National  Center  for  Advanced 
Technologies,  and  National  Defense  Industrial  Association  to  discuss 
manufacturing  topics  and  MRLs.  On  manufacturing  workforce  issues,  we 
interviewed  officials  responsible  for  planning  activities  within  each  of  the 
military  services  and  the  Defense  Management  Contract  Agency.  We 
compared  manufacturing  and  production  considerations  in  the  prior 


^AO,  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs,  GAO-09-326SP 
(Washington,  D.C.:  Mar.  30,  2009). 
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version  of  DOD’s  policy  instruction  on  operation  of  the  defense  acquisition 
system2  to  those  in  the  current  version  of  the  policy  instruction.3 

To  identify  manufacturing  best  practices  of  leading  commercial 
companies,  we  interviewed  and  obtained  documentation  from 
manufacturing,  quality,  and  supplier  personnel  at  five  companies,  and 
reported  on  four  companies:  GE  Aviation,  an  aerospace  company;  GE 
Healthcare,  a  producer  of  healthcare  products  and  services;  Honeywell 
Aerospace,  a  provider  of  aircraft  integrated  avionics,  engines,  systems,  and 
services;  Siemens  Mobility,  a  producer  of  light  rail  cars.  We  selected 
companies  that  manufacture  complex  products  and  have  won  awards  for 
quality  manufacturing.  Appendix  I  includes  additional  details  about  our 
scope  and  methodology.  We  conducted  this  performance  audit  from 
January  2009  to  February  2010  in  accordance  with  generally  accepted 
government  auditing  standards.  These  standards  require  that  we  plan  and 
perform  the  audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a 
reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit 
objectives. 

We  are  making  recommendations  to  the  Secretary  of  Defense  to  require  an 
assessment  of  the  manufacturing  readiness  across  DOD  programs  using 
MRL  criteria,  examine  strengthening  the  MRL  criteria  related  to  process 
capability  and  control,  assess  analytical  model  needs  and  tools  to  support 
MRL  assessments,  and  assess  the  manufacturing  workforce  knowledge 
and  skills  base  and  develop  a  plan  to  address  DOD’s  current  and  future 
workforce  knowledge  gaps.  In  commenting  on  a  draft  of  this  report,  DOD 
partially  concurred  with  the  first  recommendation,  and  concurred  with  the 
other  recommendations. 


Background 


In  recognition  of  the  lack  of  manufacturing  knowledge  at  key  decision 
points  and  the  need  to  develop  more  affordable  weapon  systems,  DOD 
made  recent  changes  to  its  policy.  In  2008,  the  department  made 
constructive  changes  to  its  policy  instruction  on  operation  of  the  defense 
acquisition  system.  It  also  developed  MRLs  as  a  measure  that  could 
strengthen  the  way  the  department  manages  and  develops  manufacturing¬ 
intensive  systems.  In  2004,  the  Joint  Defense  Manufacturing  Technology 


department  of  Defense  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System 
(May  12,  2003). 

department  of  Defense  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System 
(Dec.  8,  2008).  (Hereafter  cited  as  DODI  5000.02  (Dec.  8,  2008)). 
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Panel4  sponsored  a  joint  defense  and  industry  working  group  to  design  and 
develop  MRLs  for  programs  across  DOD.  In  May  2005,  MRLs  were  first 
introduced  to  the  defense  community  in  DOD’s  Technology  Readiness 
Assessment  Deskbook  for  science  and  technology  and  acquisition 
managers  to  consider. 

As  new  manufacturing  readiness  criteria,  MRLs  are  a  measurement  scale 
designed  to  provide  a  common  metric  and  vocabulary  for  assessing 
manufacturing  maturity  and  risk.  MRL  assessments  identify  the  risks  and 
manufacturing  readiness  of  a  particular  technology,  manufacturing 
process,  weapon  system,  subsystem,  or  element  of  a  legacy  program  at  key 
milestones  throughout  the  acquisition  life  cycle.  There  are  10  basic  MRLs 
designed  to  be  roughly  congruent  with  comparable  levels  of  technology 
readiness  levels  for  ease  of  use  and  understanding.  Table  1  shows  the 
MRLs  and  basic  definitions  (see  appendix  II  for  the  detailed  MRL 
definitions). 


Table  1 : 

Basic  Manufacturing  Readiness  Level  Definitions 

MRL 

Description 

1 

Basic  manufacturing  implications  identified 

2 

Manufacturing  concepts  identified 

3 

Manufacturing  proof  of  concept  developed 

4 

Capability  to  produce  the  technology  in  a  laboratory  environment 

5 

Capability  to  produce  prototype  components  in  a  production-relevant  environment 

6 

Capability  to  produce  a  prototype  system  or  subsystem  in  a  production-relevant  environment 

7 

Capability  to  produce  systems,  subsystems,  or  components  in  a  production-representative  environment 

8 

Pilot  line  capability  demonstrated;  ready  to  begin  low-rate  initial  production 

9 

Low-rate  production  demonstrated;  capability  in  place  to  begin  full-rate  production 

10 

Full-rate  production  demonstrated,  and  lean  production  practices  in  place 

Source:  J  oint  Defense  Manufacturing  Technology  Panel. 


40n  June  8,  1999,  the  Joint  Defense  Manufacturing  Technology  Panel  was  chartered  by  the 
Office  of  the  Director,  Defense  Research  &  Engineering,  the  military  services,  and  the 
Defense  Logistics  Support  Command  to  conduct  joint  program  planning.  The  Joint  Defense 
Manufacturing  Technology  Panel  develops  joint  strategies  for  the  Manufacturing 
Technology  programs  conducted  by  the  Army,  Navy,  Air  Force,  and  Defense  Logistics 
Agency  (the  components).  Other  duties  include,  but  are  not  limited  to  (1)  reviews  and 
assessments  of  defense-related  manufacturing  issues;  (2)  annual  planning  activities  with 
the  Office  of  the  Director  of  Defense  Research  &  Engineering;  and  (3)  information 
exchange  with  government  agencies,  private  industry,  academia,  and  professional 
associations — conducting  and  supporting  an  annual  Defense  Manufacturing  Conference. 
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The  working  group  also  developed  a  set  of  elements  called  “threads”  to 
provide  acquisition  managers  and  those  conducting  assessments  an 
understanding  of  the  manufacturing  risk  areas  (see  table  2).  For  these 
threads,  desired  progress  is  defined  for  each  MRL,  to  provide  an 
understanding  of  risks  as  readiness  levels  increase  from  one  MRL  to  the 
next.  Conceptually,  these  threads  are  manufacturing  elements  that  are 
essential  to  programs  as  they  plan,  prepare  for,  and  manage  the  activities 
necessary  to  develop  a  product.  For  example,  the  materials  thread  requires 
an  assessment  of  potential  supplier  capability  by  MRL  3  and  an  assessment 
of  critical  first-tier  suppliers  by  MRL  7.  Likewise,  the  manufacturing 
personnel  thread  calls  for  identifying  new  manufacturing  skills  by  MRL  3 
and  identifying  manufacturing  workforce  requirements  for  the  pilot  line  by 
MRL  7. 


Table  2:  Basic  Manufacturing  Threads  (Risk  Areas)  for  MRL  1-10 

Thread  (risk  areas) 

Description 

Technology  and  the  Industrial  Base 

Requires  an  analysis  of  the  capability  of  the  national  technology  and  industrial  base  to  support 
the  design,  development,  production,  operation,  uninterrupted  maintenance  support  of  the 
system  and  eventual  disposal  (environmental  impacts). 

Design 

Requires  an  understanding  of  the  maturity  and  stability  of  the  evolving  system  design  and  any 
related  impact  on  manufacturing  readiness. 

Costand  Funding 

Requires  an  analysis  of  the  adequacy  of  funding  to  achieve  target  manufacturing  maturity 
levels.  Examines  the  risk  associated  with  reaching  manufacturing  cost  targets. 

Materials 

Requires  an  analysis  of  the  risks  associated  with  materials  (including  basic/raw  materials, 
components,  semifinished  parts,  and  subassemblies). 

Process  Capability  and  Control 

Requires  an  analysis  of  the  risks  that  the  manufacturing  processes  are  able  to  reflect  the 
design  intent  (repeatability  and  affordability)  of  key  characteristics. 

Quality  Management 

Requires  an  analysis  of  the  risks  and  management  efforts  to  control  quality,  and  foster 
continuous  improvement. 

Manufacturing  Personnel 

Requires  an  assessment  of  the  required  skills,  availability,  and  required  number  of  personnel 
to  support  the  manufacturing  effort. 

Facilities 

Requires  an  analysis  of  the  capabilities  and  capacity  of  key  manufacturing  facilities  (prime, 
subcontractor,  supplier,  vendor,  and  maintenance/repair). 

Manufacturing  Management 

Requires  an  analysis  of  the  orchestration  of  all  elements  needed  to  translate  the  design  into 
an  integrated  and  fielded  system  (meeting  program  goals  for  affordability  and  availability). 

Source:  J  oint  Defense  Manufacturing  Technology  Panel 


As  shown,  each  basic  thread  (risk  area)  has  a  description  and  general 
requirements  for  assessing  risks  for  each  thread.  The  working  group 
further  decomposed  these  MRL  threads  into  subthreads  to  provide  users  a 
detailed  understanding  of  the  various  kinds  of  manufacturing  risks.  See 
appendix  III  for  a  detailed  breakdown  of  these  threads  (risk  areas)  for 
each  MRL. 
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DOD’s  Long-standing 
History  of  Manufacturing 
Problems 


GAO  has  conducted  an  extensive  body  of  work  that  highlights  many  of  the 
manufacturing-related  problems  that  both  DOD  and  its  prime  contractors 
have  faced.  In  many  respects,  DOD  has  recognized  the  nature  of  these 
problems  throughout  the  years  and  has  taken  a  number  of  proactive  steps 
to  address  them.  GAO’s  work  has  drawn  on  lessons  learned  and  best 
practices  to  recommend  ways  for  DOD  to  improve  the  way  it  develops  and 
manufactures  its  weapon  systems.  Examples  from  our  reports  include  the 
following: 


•  In  1996,  GAO  reported  the  practices  that  world-class  commercial 
organizations  had  adopted  to  more  efficiently  produce  quality 
products,  to  improve  DOD’s  quality  assurance  program.5  DOD  was 
spending  $1.5  billion  extra  per  year  on  military-unique  quality 
assurance  requirements  for  major  acquisitions  and  billions  more  on 
cost  and  schedule  overruns  to  correct  problems.  GAO  concluded  that 
repeated  unstable  designs,  poor  process  controls,  and  poor  transition 
to  production  caused  the  manufacturing  quality  problems.  While  DOD 
had  taken  some  actions,  its  culture  was  cited  as  the  biggest  reason  for 
slow  adoption  and  unimplemented  recommendations. 

•  In  1998,  GAO  reported  on  best  commercial  practices  to  offer  ways  to 
improve  the  process  DOD  uses  to  manage  suppliers  engaged  in 
developing  and  producing  major  weapon  systems.6  In  assessing 
defense  contractors  and  two  case  studies  of  munitions  programs,  the 
report  concluded  that  suppliers  were  critical  in  the  amount  of 
technological  innovation  they  contribute  to  the  final  product. 

•  In  2002,  GAO  reported  on  how  best  practices  could  offer 
improvements  to  the  way  DOD  develops  new  weapon  systems, 
primarily  the  design  and  manufacturing  aspects  of  the  acquisition 
process.7  DOD’s  record  showed  a  history  of  taking  longer  and 
spending  more  than  planned  to  develop  and  acquire  weapon  systems, 
which  reduced  its  buying  power.  The  report  identified  and 
recommended  best  practices  for  capturing  and  using  design  and 
manufacturing  knowledge  early  and  new  development  processes  that 


5GAO ,  Best  Practices:  Commercial  Quality  Assurance  Practices  Offer  Improvements  for 
DOD,  GAO/NSIAD-96-162  (Washington,  D.C.:  Aug.  26,  1996). 

6GAO,  Best  Practices:  DOD  Can  Help  Suppliers  Contribute  More  to  Weapon  System 
Programs,  GAO/NSIAD-98-87  (Washington,  D.C.:  Mar.  17,  1998). 

7GAO,  Best  Practices:  Capturing  Design  and  Manufacturing  Knowledge  Early  Improves 
Acquisition  Outcomes,  GAO-02-701  (Washington,  D.C.:  July  16,  2002). 
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included  high-level  decision  points  and  knowledge-based  exit  criteria 
before  key  decisions  on  production  are  made.  Essentially,  one  of  the 
high-level  decision  points  has  become  what  GAO  commonly  refers  to 
as  Knowledge  Point  3 — the  point  when  a  program  has  demonstrated 
the  manufacturing  processes  are  mature.  The  report  also 
recommended  a  best  practice  that  includes  a  standard  called  the 
Process  Capability  Index  (Cpk),  a  process  performance  measurement 
that  quantifies  how  closely  a  product  is  running  to  its  specification 
limits.  The  index  indicates  how  well  the  processes  statistical 
performance  meets  its  control  limit  requirement. 

•  In  2008,  GAO  reported  on  how  DOD  and  its  defense  contractors  can 
improve  the  quality  of  major  weapon  systems.8  We  reported  that  if 
DOD  continued  to  employ  the  same  acquisition  practices  as  it  has  in 
the  past,  the  cost  of  designing  and  developing  its  systems  could 
continue  to  exceed  estimates  by  billions  of  dollars.  Quality  problems 
were  identified  as  the  cause  for  cost  overruns,  schedule  delays,  and 
reduced  weapon-system  availability.  Like  DOD  prime  contractors, 
leading  commercial  firms  rely  on  many  practices  related  to  systems 
engineering,  manufacturing,  and  supplier  quality,  but  they  were  more 
disciplined,  and  had  institutionalized  processes  to  ensure  quality. 

•  Since  2003,  GAO  has  issued  a  series  of  annual  assessment  reports  on 
selected  weapons  programs,  increasing  from  77  to  96  programs 
reviewed.9  At  $296  billion,  the  cumulative  cost  growth  for  DOD 
programs  reported  in  2009  was  found  to  be  higher  than  it  had  been  five 
years  earlier,  and  the  overall  performance  of  weapon  system  programs 
was  still  poor.  Although  the  cost  growth  and  the  22-month  average 
delay  in  delivering  initial  capabilities  was  not  attributed  to 
manufacturing  alone,  the  lack  of  production  maturity  was  cited  as  one 
of  three  key  knowledge  areas  contributing  to  the  department’s  cost 
growth,  schedule  delay,  and  performance  problems. 


8GAO,  Best  Practices:  Increased  Focus  on  Requirements  and  Oversight  Needed  to 
Improve  DOD’s  Acquisition  Environment  and  Weapon  System  Quality,  GAO-08-294 
(Washington,  D.C.:  Feb.  1,  2008). 

9GAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs,  GAO-09-326SP 
(Washington,  D.C.:  Mar.  30,  2009). 
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Revised  Policy 
Incorporates 
Manufacturing  Best 
Practices 


DOD’s  December  2008  revision  to  its  policy  instruction  on  operation  of  the 
defense  acquisition  system10  incorporates  a  number  of  the  best  practices 
we  identified  in  our  previous  work.  The  instruction  covers  the  entire  life 
cycle  and  considers  manufacturing  risks  earlier  in  the  acquisition  life-cycle 
framework.  In  a  November  2003  report  on  DOD’s  May  2003  revision  to  its 
policy,  we  reported  that  much  of  the  revised  policy  agrees  with  GAO’s 
extensive  body  of  work  and  that  of  successful  commercial  firms.  While  we 
assessed  DOD’s  revised  policy  as  providing  a  good  framework  for 
capturing  knowledge  about  critical  technologies,  product  design  and 
manufacturing  processes,  we  reported  in  2006  that  acquisition  officials 
were  not  effectively  implementing  the  acquisition  policy’s  knowledge- 
based  process. 11  We  reported  that  the  effective  implementation  of  policy 
was  limited  by  the  absence  of  effective  controls  that  require  compliance 
and  specific  criteria  for  clearly  demonstrating  that  acceptable  levels  of 
knowledge  about  technology,  design,  and  manufacturing  have  been 
attained  at  critical  junctures  before  making  further  investments  in  a 
program.  We  concluded  that  without  specific  criteria — or  standards 
against  which  a  judgment  or  decision  is  quantifiably  based — decision 
makers  are  permitted  to  make  decisions  on  the  basis  of  subjective 
judgment.  The  December  2008  revised  policy  instruction  establishes  target 
maturity  criteria  for  measuring  risks  associated  with  manufacturing 
processes  at  milestone  decision  points.12 

During  the  material  solutions  phase,  prior  to  milestone  A,  the  2008  policy 
instruction  requires  the  analysis  of  alternatives  to  assess  “manufacturing 
feasibility.”  During  the  technology  development  phase,  prior  to  milestone 
B,  the  instruction  states  the  following: 

•  Prototype  systems  or  appropriate  component-level  prototyping  shall 
be  employed  to  “evaluate  manufacturing  processes.” 


10DODI  5000.02  (Dec.  8,  2008). 

nGAO,  Defense  Acquisitions:  Major  Weapon  Systems  Continue  to  Experience  Cost  and 
Schedule  Problems  under  DOD’s  Revised  Policy,  GAO-06-368  (Washington,  D.C.:  Apr.  13, 
2006). 

12The  three  decision  points  within  the  DOD  acquisition  framework  include:  milestone  A 
(entry  point  for  the  technology  development  acquisition  phase);  milestone  B  (entry  point 
for  the  engineering  and  manufacturing  development  period — which  is  comprised  of  two 
phases  called  integrated  system  design,  and  system  capability  and  manufacturing  process 
demonstration);  and  milestone  C  (entry  point  for  the  production  and  deployment 
acquisition  phase).  DODI  5000.02  (Dec.  8,  2008),  enc.  2,  fig.  1. 
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•  A  successful  preliminary  design  review  will  “identify  remaining  design, 
integration,  and  manufacturing  risks.” 

•  A  program  may  exit  the  technology  development  phase  when  “the 
technology  and  manufacturing  processes  for  that  program  or 
increment  have  been  assessed  and  demonstrated  in  a  relevant 
environment”  and  “manufacturing  risks  have  been  identified.” 

After  milestone  B,  one  of  the  purposes  of  the  engineering  and 
manufacturing  development  phase  is  to  “develop  an  affordable  and 
executable  manufacturing  process.”  The  instruction  says  that:  “the 
maturity  of  critical  manufacturing  processes”  is  to  be  described  in  a  post- 
critical  design  review  assessment;  system  capability  and  manufacturing 
process  demonstration  shall  show  “that  system  production  can  be 
supported  by  demonstrated  manufacturing  processes;”  and  the  system 
capability  and  manufacturing  process  demonstration  effort  shall  end, 
among  other  things,  when  “manufacturing  processes  have  been  effectively 
demonstrated  in  a  pilot  line  environment,  prior  to  milestone  C.” 

Finally,  at  milestone  C,  the  instruction  establishes  two  entrance  criteria  for 
the  production  and  deployment  phase,  which  include  “no  significant 
manufacturing  risks”  and  “manufacturing  processes  [are]  under  control  (if 
Milestone  C  is  full-rate  production).”  Low-rate  initial  production  follows  in 
order  to  ensure  an  “adequate  and  efficient  manufacturing  capability.”  In 
order  to  receive  full-rate  production  approval,  the  following  must  be 
shown: 

1.  “demonstrated  control  of  the  manufacturing  process,” 

2.  “the  collection  of  statistical  process  control  data,”  and 

3.  “demonstrated  control  and  capability  of  other  critical  processes.” 

Even  with  the  updated  policy  instruction  in  place  that  includes  guidance 
for  most  knowledge-based  practices,  inconsistent  implementation  has 
hindered  DOD’s  past  efforts  to  reform  its  acquisition  practices.  For 
example,  we  reported  in  2006  that  DOD  was  not  effectively  implementing 
the  knowledge-based  approach  process  and  evolutionary  approach 
emphasized  in  its  policy.13  While  the  policy  outlined  specific  knowledge- 


13GAO,  Defense  Acquisitions:  Major  Weapon  Systems  Continue  to  Experience  Cost  and 
Schedule  Problems  under  DOD’s  Revised  Policy,  GAO-06-368  (Washington,  D.C.:  Apr.  13, 
2006). 
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based  process  of  concept  refinement  and  technology  development  to  help 
ensure  a  sound  business  case  is  developed  before  committing  to  a  new 
development  program,  we  found  that  almost  80  percent  of  the  programs 
we  reviewed  were  permitted  to  bypass  this  process. 


Manufacturing 
Problems  Are 
Attributed  to  Several 
Factors  during  the 
Planning  and  Design 
Phases  of  Selected 
DOD  Weapons 
Programs 


Defense  acquisition  programs  continue  to  have  problems  manufacturing 
weapon  systems.  As  a  result,  systems  cost  far  more  and  take  far  longer  to 
produce  than  estimated.  Many  programs  authorized  to  enter  production 
experienced  billions  of  dollars  in  cost  growth  after  the  authorization — 
nearly  two-thirds  of  those  programs  reported  increases  in  average 
procurement  unit  costs.  Several  factors  contribute  to  these  issues  during 
the  planning  and  design  phases.  These  include  the  inattention  to 
manufacturing  during  planning  and  design,  poor  supplier  management, 
and  lack  of  a  knowledgeable  manufacturing  workforce.  Essentially,  some 
of  these  programs  moved  into  production  without  considering 
manufacturing  risks  earlier  in  development.  This  hindered  managers  from 
later  managing  those  risks  until  they  became  problematic,  and  also  led  to 
subsequent  problems  with  supplier  management,  such  as  prime 
contractors  conducting  little  oversight  of  suppliers.  Some  programs  also 
had  an  inadequate  workforce — in  terms  of  insufficient  knowledge  and 
numbers — to  effectively  manage  and  oversee  defense  manufacturing 
efforts. 


Manufacturing  Contributed  Defense  acquisition  programs  continue  to  be  troubled  by  unstable 
to  Growth  in  Cost  and  requirements,  immature  technology,  and  a  lack  of  manufacturing 

Delays  in  Schedule  knowledge  early  in  design,  resulting  in  more  costly  products  that  take 

longer  to  produce.  Our  2009  annual  assessment  shows  that  total  research 
and  development  costs  were  42  percent  higher  than  originally  estimated. 
These  higher  costs  reflect  in  part  the  learning  that  takes  place  as 
manufacturing  processes  are  established  and  used  to  produce  the  first 
prototypes. 

Even  programs  that  have  been  authorized  to  begin  production  have 
experienced  substantial  cost  growth  after  the  production  decision. 
Production  performance  can  be  measured  by  examining  the  cost  growth 
as  expressed  in  changes  to  average  procurement  unit  cost.  This  represents 
the  value  DOD  gets  for  the  procurement  dollars  invested  in  a  certain 
program  and  shows  the  net  effect  of  procurement  cost  growth  and 
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quantity  changes.  Figure  1  shows  the  levels  of  average  procurement  unit- 
cost  growth  for  selected  major  defense  acquisition  programs.14 


Figure  1 :  Distribution  of  Average  Procurement  Unit-Cost  Growth  after  a  Production 
Decision  for  Major  Defense  Acquisition  Programs 


Note:  Data  include  all  major  defense  acquisition  programs  that  entered  production  in  fiscal  year  2000 
or  later. 

As  indicated  in  figure  1,  nearly  two-thirds  of  programs  that  entered 
production  after  2000  reported  more  than  a  5  percent  increase  in  average 
unit  cost  growth,  while  32  percent  of  programs  reported  average  unit  cost 
growth  that  ranged  from  11  percent  to  more  than  15  percent.  One  program 
reported  a  25  percent  increase  in  average  procurement  unit  cost.  Further, 
42  percent  of  those  programs  experienced  production  cost  increases  when 
procured  quantities  decreased  or  remained  the  same.  For  example,  the 
Black  Hawk  helicopter’s  2007  production  estimate  had  no  increase  in 
quantities  since  2005,  yet  its  production  cost  increased  $2.3  billion,  and 
average  procurement  unit  cost  rose  by  13  percent.  The  Joint  Air-to-Surface 
Standoff  Missile  had  an  8  percent  quantity  decrease  since  the  2004 


14Major  defense  acquisition  programs  are  those  identified  by  DOD  that  require  total 
research,  development,  test,  and  evaluation  (RDT&E)  expenditures,  including  all  planned 
increments,  of  more  than  $365  million  or  procurement  expenditures,  including  all  planned 
increments,  of  more  than  $2.19  billion  in  fiscal  year  2000  constant  dollars. 
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production  decision;  but  the  production  costs  increased  by  $561  million 
and  average  procurement  unit  cost  increased  by  25  percent. 

As  for  schedule  growth,  DOD  has  continued  to  experience  delays  in 
delivering  new  or  modified  weapon  systems  to  the  warfighter.  Over  50 
percent  of  current  programs  in  production  have  encountered  some  form  of 
delay  after  the  production  decision,  when  manufacturing  processes  should 
be  in  control.  Consequently,  warfighters  often  must  operate  costly  legacy 
systems  longer  than  expected,  find  alternatives  to  fill  capability  gaps,  or  go 
without  the  capability  altogether. 

The  four  DOD  weapon  systems  we  selected  for  in-depth  review  with 
known  cost,  schedule,  and  performance  problems  reported  several  key 
factors  that  contributed  to  manufacturing  problems.  These  include  the 
inattention  to  manufacturing  during  planning  and  design,  poor  planning 
for  supplier  management,  and  lack  of  a  knowledgeable  manufacturing 
workforce.  Capturing  critical  manufacturing  knowledge  during  the 
planning  and  design  phases  before  entering  production  helps  to  ensure 
that  a  weapon  system  will  work  as  intended  and  can  be  manufactured 
efficiently  to  meet  cost,  schedule,  and  quality  targets.  The  programs  in  our 
review  often  lacked  manufacturing  knowledge  at  key  decision  points, 
which  led  to  cost  growth  and  schedule  delays.  For  example,  the  Joint  Air- 
to-Surface  Standoff  Missile  program — an  autonomous,  air-to-ground 
missile  designed  to  destroy  high-value  targets — experienced  a  critical  unit- 
cost  breach  due  to  missile  reliability  problems  not  being  addressed  early  in 
the  design  phase.15  Also,  the  Electromagnetic  Aircraft  Launch  System — a 
new  catapult  technology  being  developed  for  the  Navy’s  newest  class  of 
aircraft  carriers — had  experienced  problems  manufacturing  compatible 
materials,  which  resulted  in  cost  growth  and  schedule  delays  and  was  the 
focus  of  recent  congressional  interest.  Figure  2  summarizes  contributing 
factors  for  manufacturing  problems  experienced  by  the  four  DOD  weapon 
systems. 


15In  April  9,  2007,  the  Secretary  of  the  Air  Force  reported  the  average  procurement  unit  cost 
increased  more  than  50  percent  above  the  initial  baseline  estimate  and  that  the  majority  of 
the  cost  growth  was  attributed  to  a  reliability  improvement  program  to  deal  with  the 
missile’s  reliability  problems,  addition  of  the  extended  range  variant,  and  reduction  in 
missile  quantities.  However,  the  program  reported  that,  even  without  the  quantities 
associated  with  the  extended  range  variant,  the  program  would  have  experienced  a 
significant  breach  to  the  original  baseline. 
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Figure  2:  Contributing  Factors  to  Manufacturing  Problems  for  Four  DOD  Case-Study  Programs 
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Source:  GAO  analysis  of  Army,  Air  Force,  Navy,  and  Missile  Defense  Agency  data.  Images:  Missile  Defense  Agency  and  Boeing 
public  Website  perGMG  program  office  (top);  CVN-21  Program  Office  050708-D-8455H-001  Washington,  D.C.  (J  uly  8,  2005)  U.S. 
Navy  graphic  (released)  (second  from  top);  USMC  Light/Attack  Helicopter  (H-1)  Program  Office,  PMA276.  (third  from  top);  Integrated 
Test  2  accomplished  December  2006  (bottom). 


As  indicated,  most  of  the  programs  had  more  than  one  major  problem 
related  to  manufacturing.  These  issues  illustrate  the  major  problems  we 
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discussed  with  defense  and  contractor  officials,  but  do  not  encompass  all 
the  manufacturing  problems  experienced  by  the  programs.  For  example,  a 
recent  Air  Force  study  reports  that  manufacturing  and  quality  assurance 
requirements  are  not  included  in  the  contracts  to  develop  weapon 
systems,  which  could  affect  the  contractor’s  approach  to  manufacturing. 
Officials  from  the  Defense  Contract  Management  Agency — a  DOD 
component  that  works  directly  with  defense  suppliers  to  ensure  that 
supplies  and  services  are  delivered  on  time,  at  projected  cost,  and  meet 
performance  requirements — also  reported  similar  contract  issues  that 
could  affect  contractor  performance  on  manufacturing. 


Manufacturing  Was 
Overlooked  during  Early 
Development 


Each  of  the  four  programs  we  examined  did  not  give  manufacturing  strong 
consideration  during  the  early  planning  and  design  phases.  Programs  were 
moved  into  production  largely  without  considering  manufacturing  risks 
earlier  in  the  acquisition  process,  as  demonstrated  by  the  experiences  of 
the  Exoatmospheric  Kill  Vehicle  and  the  H-l  helicopter  upgrade  program. 
The  Exoatmospheric  Kill  Vehicle  was  designed  to  intercept  and  destroy 
high-speed  ballistic  missile  warheads  in  mid-flight,  while  the  H-l  upgrade 
program  converts  the  attack  helicopter  and  the  utility  helicopter  to  the 
AH-1Z  and  UH-1Y  configurations,  respectively. 


The  Exoatmospheric  Kill  Vehicle  program  was  put  on  an  accelerated 
development  schedule  in  response  to  a  directive  to  develop  and  deploy,  at 
the  earliest  possible  date,  ballistic  missile  defense  drawing  on  the  best 
technologies  available.  According  to  the  contractor,  it  bypassed  some  of 
its  normal  development-review  processes  to  accelerate  delivery  of  the 
vehicle,  which  also  resulted  in  a  high  acceptance  of  manufacturing  risks 
without  sufficient  identification  and  management  of  risk-mitigation  plans. 
For  example,  the  program  went  into  production  without  completing 
qualification  testing.  In  addition,  the  contractor  continued  to  incorporate 
design  changes  while  supplier  production  was  ongoing,  resulting  in 
rework  and  disruption  to  the  production  line.  Early  lots  of  kill  vehicles 
were  built  manually  by  engineers  in  the  absence  of  automated  production 
processes,  which  caused  dissimilarities  among  vehicles  in  the  fleet  and 
will  make  refurbishments  difficult. 16 


For  several  reasons,  the  H-l  helicopter  upgrade  program  did  not  include 
manufacturing  in  the  early  phases  of  planning  and  also  proceeded  to 


ltJ Automated  production  processes  have  been  implemented  primarily  for  the  sensor,  and  for 
other  limited  activities,  but  have  not  been  implemented  across  the  entire  program. 
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production  before  its  design  was  mature,  according  to  the  contractor. 
First,  the  program  underestimated  the  complexity  of  updating  and 
remanufacturing  the  aircraft  without  historical  drawings.  The  emphasis 
was  placed  on  minimizing  development  costs  and  resources  were  not 
available  to  assess  manufacturing  challenges  early  in  the  redesign  process. 
Furthermore,  the  program  started  low-rate  production  before  completing 
operational  evaluation  testing.  As  a  result,  the  problems  uncovered  during 
testing  had  to  be  corrected  on  aircraft  that  were  on  the  assembly  line. 

Also,  constant  change  orders  and  factory  bottlenecks,  among  other 
problems,  affected  program  costs  and  schedules.  The  schedule  pressure 
allowed  little  opportunity  to  remedy  the  manufacturing  problems, 
resulting  in  more  complicated  and  expensive  fixes.  Ultimately  the 
schedule  slowed  and  the  costs  increased  to  the  point  that  the  program 
abandoned  the  remanufacturing  upgrade  and,  instead,  opted  to  purchase 
newly  manufactured  aircraft  cabins  for  the  UH-1Y  configuration. 


Poor  Planning  Led  to  Inattention  to  manufacturing  during  planning  and  design  led  to  subsequent 

Supplier  Problems  problems  with  supplier  management  in  two  major  defense  acquisition 

programs  we  reviewed.  Specifically,  the  prime  contractors  did  not  give 
adequate  attention  to  managing  their  suppliers.  For  example,  program 
officials  for  the  Joint  Air-to-Surface  Standoff  Missile  told  us  that  the 
responsibility  for  manufacturing  processes  and  discipline  shifted  in  the 
1990s  from  the  government  to  the  defense  contractors.  The  government 
started  to  rely  on  the  prime  contractor  to  ensure  quality  and  reliability, 
particularly  with  subtier  suppliers.  In  this  case,  the  program  office  told  us 
that  the  prime  contractor  for  the  missile  program  relied  on  the  subtier 
suppliers  to  self-report  their  capabilities  and  did  not  engage  in  effective 
oversight  of  their  work,  which  led  to  defective  parts.  The  program  office 
recently  recruited  experts  in  manufacturing  to  help  the  prime  contractor 
address  their  supplier  problems  more  effectively. 

In  the  Exoatmospheric  Kill  Vehicle  program,  supplier  quality  was 
inconsistent,  resulting  in  unnecessary  rework  and  uncovering  problems 
late  in  production.  For  many  suppliers,  the  kill  vehicle  program  represents 
a  small  portion  of  their  business,  so  the  emphasis  on  quality  was  often 
lacking.  Further,  the  program  was  initially  procured  as  a  capability  based 
program,  rather  than  requirements  based  program.  Thus,  the  prime 
contractor  did  not  impose  requirements  on  the  subcontractors  to  comply 
with  stringent  requirements  for  space  programs.  In  turn,  the 
subcontractors  did  not  implement  sufficient  requirements  which  led  to 
recurring  quality  issues. 
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Lack  of  Manufacturing 
Knowledge  Contributed  to 
Problems 


Some  DOD  programs  and  prime  contractors  had  an  inadequate  defense 
manufacturing  workforce — both  in  terms  of  numbers  and  experience — to 
effectively  manage  and  oversee  manufacturing  efforts,  which  resulted  in 
schedule  delays  or  cost  inefficiencies.  The  manufacturing  workforce 
includes  occupations  such  as  specialists  in  quality  assurance,  business, 
manufacturing  engineering,  industrial  engineering,  and  production  control. 
In  many  cases,  the  programs  lacked  manufacturing  expertise  early  in 
development,  which  hindered  the  program’s  ability  to  later  manage 
manufacturing  risks.  For  example,  the  contractor  for  the  Electromagnetic 
Air  Launch  System  did  not  have  sufficient  systems-engineering  personnel 
involved  in  the  design  to  help  it  transition  from  development  to 
production.  As  a  result,  the  program  encountered  schedule  delays  and  cost 
increases.  DOD  conducted  a  program  assessment  review,  which  led  the 
program  office  and  contractor  to  increase  systems  engineering  staff. 

For  the  Exoatmospheric  Kill  Vehicle  program,  the  contractor’s  workforce 
and  manufacturing  processes  could  not  readily  undertake  the  rigors  of 
production  for  a  space-based  capability,  part  of  which  must  be 
manufactured  in  a  clean  room  environment,  and  all  of  which  commands 
rigorous  processes  and  procedures  due  to  highly  technical  designs.  The 
contractor’s  hourly  assembly  personnel  were  trained  to  build  tactical 
missiles  on  a  high-rate  production  line  and  were  not  sufficiently  trained  in 
the  quality-control  standards  required  by  clean-room  manufacturing,  such 
as  carefully  controlling  foreign-object  debris,  specially  maintaining  the 
clean  room,  and  using  a  partner  in  certain  high-level  tasks  to  ensure  all 
steps  are  properly  followed.  These  standards  were  not  institutionalized, 
and  the  contractor  eventually  had  to  modify  its  facilities  and  production 
standards  to  correct  the  manufacturing  problems.  The  facility  had  to  be 
retooled  and  reconfigured  late  in  development.  The  contractor  also 
experienced  high  turnover  in  its  workforce  due  to  the  increasing  demands 
associated  with  working  in  a  clean-room  environment  and  working  long 
hours. 
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MRLs  Have  Been 
Proposed  to  Improve 
the  Way  DOD 
Identifies  and 
Manages 

Manufacturing  Risk 
and  Readiness 


The  Joint  Defense  Manufacturing  Technology  Panel  working  group  has 
proposed  MRLs  as  new  manufacturing  readiness  criteria  that  could 
improve  weapon  system  outcomes  by  standardizing  the  way  programs 
identify  and  manage  manufacturing  risks  associated  with  developing  and 
fielding  advanced  weapon  systems.  MRLs  were  first  introduced  to  the 
defense  community  in  DOD’s  2005  Technology  Readiness  Assessment 
Deskbook  as  an  important  activity  for  science  and  technology  and 
acquisition  managers  to  consider.  An  analysis  by  the  working  group  shows 
that  MRLs  address  many  of  the  manufacturing  issues  not  covered  by 
DOD’s  technical  reviews,  particularly  reviews  conducted  in  the  early 
phases  of  acquisition.  In  their  development,  comprehensive  efforts  were 
undertaken  to  design  and  develop  MRLs  from  DOD  as  well  as  industry 
resources.  For  example,  the  working  group  formulated  MRLs  from  a 
manufacturing  knowledge  base  of  defense,  industry,  and  academia  to 
address  two  key  areas  of  risk — immature  product  technologies  and 
immature  manufacturing  capability.  The  working  group  also  designed 
MRLs  as  a  structured  and  disciplined  approach  for  the  way  manufacturing 
risk  and  readiness  is  expected  to  be  identified  and  assessed.  The  working 
group  also  developed  a  set  of  tools  that  include  a  deskbook,  checklist,  and 
a  website  to  help  managers  and  users  apply  MRLs  and  conduct 
assessments.  In  addition,  the  Army  and  Air  Force  report  that  their  use  of 
MRLs  on  pilot  programs  contributed  to  substantial  cost  benefits  on  a 
variety  of  programs,  including  major  acquisition  programs. 


MRLs  Were  Developed 
from  Knowledge-Based 
Resources  on 
Manufacturing 


To  develop  MRLs,  the  working  group  conducted  comprehensive  sessions 
with  industry  participants  to  ensure  the  metrics  and  vocabulary  for 
assessing  manufacturing  readiness  would  be  an  all-inclusive  body  of 
knowledge.  Officials  stated  that  a  mature  set  of  manufacturing  knowledge 
resources  already  existed  but  it  was  scattered  and  not  consistently  applied 
in  a  disciplined  way  that  aligned  with  the  DOD  acquisition  life-cycle 
framework.  In  their  formulation,  MRLs  were  developed  from  an  extensive 
body  of  manufacturing  knowledge  that  included,  but  was  not  limited  to, 
the  following  defense,  industry,  and  academic  sources: 

•  DOD  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System 
(Dec.  8,  2008), 

•  Navy  best-practices  manual  for  using  templates  on  design  and 
manufacturing  best  practices, 

•  Air  Force  manufacturing  development  guide, 

•  military  standards  and  specifications,  and 
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•  Malcolm  Baldrige  quality  award  criteria. 

Other  standards  and  technical  sources  were  obtained  from  the  Institute  of 
Electrical  and  Electronics  Engineers,  the  International  Standards 
Organization  on  quality  management  systems,  automotive  industry  quality 
standards,  and  the  supplier  model  from  the  Massachusetts  Institute  of 
Technology. 


Analysis  Shows  MRLs 
Address  Manufacturing 
Gaps  in  DOD’s  Technical 
Reviews 


An  analysis  conducted  by  the  working  group  shows  that  MRLs  address 
many  of  the  manufacturing  gaps  identified  in  several  of  DOD’s  technical 
reviews17  that  provide  program  oversight  and  determine  how  well 
programs  are  meeting  expected  goals,  particularly  the  reviews  conducted 
in  the  early  acquisition  phases.  According  to  the  working  group, 
addressing  these  manufacturing  gaps  is  fundamental  to  improving  the  way 
programs  plan,  design,  and  prepare  for  manufacturing.  For  example,  the 
working  group’s  analysis  shows  that  DOD’s  current  systems-engineering 
technical  review  checklist  used  for  preliminary  design  reviews18  has  only 
27  of  759  total  questions  that  deal  with  core  manufacturing-related 
questions,  whereas  the  MRL  6  assessment  checklist  for  this  juncture  has 
169  core  manufacturing  questions.  More  importantly,  the  technical  review 
checklist  did  not  address  key  manufacturing  discipline  in  the  areas  of 
program  management,  systems  engineering,  requirements  management, 
risk  management,  and  program  schedule.  Similarly,  the  technical  review 
checklist  used  for  critical  design  reviews19  has  only  22  of  824  total 
questions  that  deal  with  core  manufacturing  questions,  whereas  the  MRL  7 
assessment  checklist  for  this  juncture  has  162  core  questions.  Core 
manufacturing  disciplines  were  not  addressed  in  the  specific  areas  of 
management  metrics,  manufacturing  planning,  requirements  management, 
system  verification,  and  other  areas.  Finally,  DOD’s  technical  review 


17Technical  reviews  fall  under  the  Systems  and  Software  Engineering  organization,  within 
the  Office  of  the  Secretary  of  Defense,  where  responsibilities  for  quality  and  oversight  of 
defense  programs  reside. 

18The  preliminary  design  review  ensures  that  the  system  under  review  has  a  reasonable 
expectation  of  satisfying  the  requirements  within  the  currently  allocated  budget  and 
schedule.  The  review  includes  evaluation  areas  and  checklist  questions. 

19The  critical  design  review  ensures  that  the  system  under  review  has  a  reasonable 
expectation  of  satisfying  the  requirements  of  the  Capability  Development  Document  within 
the  currently  allocated  budget  and  schedule.  This  review  assesses  the  final  design  as 
captured  in  product  specifications  for  each  configuration  item  in  the  system  and  ensures 
that  each  product  specification  has  been  captured  in  detailed  design  documentation. 


Page  18 


GAO-10-439  DOD  Manufacturing  Readiness 


checklist  used  for  production  readiness  reviews30  has  194  of  613  total 
questions  that  deal  with  core  manufacturing  questions.  While  the  MRL  8 
assessment  checklist  has  14  fewer  core  questions  on  manufacturing  at  this 
juncture,  the  working  group  stated  these  core  manufacturing  questions  are 
addressed  earlier  in  the  acquisition  framework,  which  is  reflective  of 
commercial  best  practices  where  such  manufacturing  topics  and  discipline 
are  addressed,  in  contrast  to  DOD’s  current  practice. 


Draft  Deskbook  Explains  The  draft  MRL  deskbook  is  a  detailed  instructional  resource  on  how  to 

MRL  Application  and  apply  MRLs  and  conduct  assessments  of  manufacturing  risk  and  readiness, 

Assessments  such  as  how  to  structure  and  apply  evaluations  to  a  technology, 

component,  manufacturing  process,  weapon  system,  or  subsystem  using 
the  MRL  definitions.  It  also  demonstrates  how  assessments  should  be 
carried  out  at  various  phases  by  the  managers  of  science  and  technology 
projects  and  technology  demonstration  projects  intending  to  transition 
directly  to  the  acquisition  community,  as  well  as  acquisition  program 
managers  and  the  people  involved  in  conducting  assessments.  According 
to  the  working  group,  MRLs  can  not  only  be  used  to  improve  how  DOD 
manages  and  communicates  manufacturing  risk  and  readiness,  but  can 
also  give  decision  makers  and  manager’s  better  visibility  into  program 
risks.  For  example,  a  variety  of  manufacturing  status  and  risk  evaluations 
have  been  performed  for  years  as  part  of  defense  acquisition  programs  in  a 
variety  of  forms — for  example,  production  readiness  reviews, 
manufacturing  management/production  capability  reviews,  etc.  However, 
these  structured  and  managed  reviews  do  not  use  a  uniform  metric  to 
measure  and  communicate  manufacturing  risk  and  readiness. 

MRLs,  when  used  in  combination  with  technology  readiness  levels,  are 
expected  to  address  two  key  risk  areas — immature  product  technologies 
and  immature  manufacturing  capability.  The  draft  deskbook  says  that  it  is 
common  for  manufacturing  readiness  to  be  paced  by  technology  readiness 
or  design  stability,  and  that  it  is  not  until  the  product  technology  and 
product  design  are  stable  that  manufacturing  processes  will  be  able  to 
mature.  MRLs  can  also  be  used  to  define  manufacturing  readiness  and  risk 


20The  production  readiness  review  examines  a  program  to  determine  if  the  design  is  ready 
for  production  and  if  the  prime  contractor  and  major  subcontractors  have  accomplished 
adequate  production  planning  without  incurring  unacceptable  risks  that  will  breach 
thresholds  of  schedule,  performance,  cost,  or  other  established  criteria.  The  review 
evaluates  the  full,  production-configured  system  to  determine  if  it  correctly  and  completely 
implements  all  system  requirements. 
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at  the  system  or  subsystem  level.  For  these  reasons,  the  MRL  definitions 
were  designed  to  include  a  target  level  of  technology  readiness  as  a 
prerequisite  for  each  level  of  manufacturing  readiness.  Figure  3  shows  the 
relationship  of  MRLs  to  system  milestones  and  technology  readiness  levels 
in  the  defense  acquisition  life-cycle  framework. 


Figure  3:  Relationship  of  MRLs  to  System  Milestones  and  Technology  Readiness  Levels  (TRL) 
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Note:  Alignment  of  MRLs  and  TRLs  within  the  DOD  acquisition  framework  are  generalized  and  may 
not  align  exactly  as  illustrated. 
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MRL  Assessments  Provide 
Basis  for  Identifying, 
Planning,  and  Managing 
Program  Risks 


MRL  assessments  are  intended  to  leverage  better  manufacturing 
knowledge,  enabling  managers  to  be  aware  of  problems  or  risks  early  in 
development,  when  they  are  easier  to  resolve  and  before  significant 
investments  are  made.  In  turn,  these  risks  can  be  addressed  earlier  in  the 
life  cycle  when  costs  are  lower.  For  example,  the  ability  to  transition 
technology  smoothly  and  efficiently  from  the  laboratories,  onto  the  factory 
floor,  and  into  the  field  is  a  critical  enabler  for  evolutionary  acquisition. 


Assessments  can  be  applied  to  a  technology,  manufacturing  process, 
weapon  system,  or  subsystem  using  the  definitions  as  a  standard.  As  part 
of  the  assessment,  a  comparison  is  made  between  the  actual  MRLs  and  the 
target  MRL  levels.  The  difference  between  the  two  identifies  the  risks  and 
forms  the  basis  for  assisting  managers  to  develop  a  plan — called  a 
manufacturing  maturation  plan — to  remove  or  reduce  them.  Risks  should 
be  identified  throughout  the  life  cycle  and,  when  targets  are  not  met,  the 
plan  updated  to  ensure  the  appropriate  MRL  will  be  achieved  at  the  next 
decision  point.  The  manufacturing  maturation  plan  identifies 
manufacturing  risks  and  provides  a  plan  for  mitigating  each  risk  area 
throughout  the  duration  of  the  technology  or  product-development 
program.  The  draft  MRL  deskbook  says  every  assessment  of 
manufacturing  readiness  should  have  an  associated  plan  for  areas  where 
the  MRL  has  not  achieved  its  target  level.  The  deskbook  requires  a 
manufacturing  maturation  plan  to  include  the  most  essential  items  in 
planning  for  the  maturity  of  an  element  of  assessment  that  is  below  its 
target  MRL.  These  include  a  statement  of  the  problem  that  describes  areas 
where  manufacturing  readiness  falls  short  of  the  target  MRLs,  including 
key  factors  and  driving  issues,  solution  options  and  consequences  of  each 
option,  and  a  maturation  plan  with  a  schedule  and  funding  breakout.  Other 
information  should  include  the  status  of  funding  to  execute  the 
manufacturing  plan  and  specific  actions  to  be  taken  and  by  whom,  and  the 
MRL  to  be  achieved  and  when  it  will  be  achieved. 


MRL  Pilot  Programs  Show  Army  and  Air  Force  programs  have  pilot-tested  MRLs  on  science  and 
Positive  Benefits  technology  and  some  major  acquisition  programs  in  an  effort  to  increase 

the  manufacturing  readiness  and  maturity  to  higher  levels  appropriate  to 
the  phase  of  development.  Both  services  performed  MRL  assessments  on 
selected  pilot  programs  to  address  manufacturing  risks  and  assess 
technology  transition.  The  Army  reports  numerous  benefits  from  the  use 
of  MRLs  such  as  manufacturing  efficiencies,  improved  labor  utilization, 
and  cost  benefits.  Similarly,  the  Air  Force  has  used  MRLs  to  manage  its 
manufacturing  risks  associated  with  new  technologies,  yielding  tangible 
benefits.  While  MRLs  cannot  take  full  credit  for  all  benefits  derived  in  the 
pilot  programs,  officials  noted  they  are  a  good  way  to  manage,  mitigate, 
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and  communicate — between  science  and  technology,  acquisition,  the  user, 
and  the  system  developer — readiness  and  risks  early  and  throughout  the 
acquisition  process  to  avoid  major  consequences  from  manufacturing- 
related  problems.  These  programs  provide  insight  on  how  the  acquisition 
community  can  utilize  MRLs  within  weapon  system  programs. 

Army  In  2004,  the  Army’s  Aviation  and  Missile  Research,  Development  and 

Engineering  Center  began  applying  MRLs  to  various  technologies  in 
concept  development,  including  those  technologies  transitioning  to 
engineering  and  manufacturing  development.  Officials  stated  that  without 
cost  and  manufacturing  readiness  planning,  science  and  technology 
programs  face  certain  barriers  to  transition,  resulting  in:  (1)  high  unit 
production  cost  caused  by  a  focus  on  technology  without  regard  to 
affordability;  and  (2)  manufacturing  problems  caused  by  design 
complexity  resulting  in  a  technology  that  is  not  feasible  to  manufacture. 
For  example,  the  Army  has  applied  MRLs  to  many  programs,  including 
warfighter-protection  materials,  Micro-Electro-Mechanical  Systems, 
embedded  sensors,  and  helicopter  cabin  structures.  The  warfighter- 
protection  program — the  next  generation  of  helmets  and  body  gear — 
reported  that  it  was  able  to  reduce  scrap  by  60  percent  and  reduced  touch 
labor  by  20  to  40  percent.  On  programs  where  cost  benefits  could  be 
roughly  calculated,  the  Army  believes  that  MRLs,  among  other 
improvement  initiatives,  contributed  to  the  $426  million  in  benefits  on 
seven  programs.  MRLs  were  also  used  as  a  metric  in  the  Technology 
Transition  Agreement  to  communicate  manufacturing  maturity  and 
facilitate  a  smooth  transition  to  the  acquisition  community. 

Air  Force  Mr  Force  officials  we  met  with  discussed  using  MRLs  to  assess  and 

identify  gaps  and  understand  risks  in  manufacturing  maturity  that  would 
delay  technology  transition  into  an  advanced  systems  development 
program  or  a  fielded  system  upgrade.  The  Air  Force  has  conducted  several 
MRL  assessments  on  advanced  technology  demonstrations  and  major 
defense  acquisition  programs,  including  the  MQ-9  Reaper  Unmanned 
Aircraft,  Joint  Strike  Fighter,  Advance  Medium-Range  Air-to-Air  Missile, 
X-band  thin  radar  array,  and  Sensor  Hardening  for  Tactical  Systems. 
Officials  reported  that  the  use  of  MRLs  have  contributed  millions  of 
dollars  in  cost  avoidance,  increased  production  rates,  and  has  accelerated 
technology  transition.  For  example,  the  Air  Force  reported  realizing  $65 
million  in  savings  by  addressing  problems  with  a  costly  manual  drilling 
process.  MRLs  were  used  to  raise  new  drilling  technology  from  MRL  4  to 
MRL  9,  achieving  a  unit-cost  savings  of  $17,000  per  aircraft  from  reduced 
tooling,  manpower,  floor  space  usage,  and  time. 
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Because  of  MRL  assessment’s  success  on  advanced  technology  programs, 
the  Assistant  Secretary  of  the  Air  Force  for  Acquisition  directed  the 
program  office  to  perform  MRL  assessments  on  key  MQ-9  Reaper 
manufacturing  processes  and  technologies.  The  MQ-9  Reaper  is  an 
unmanned  aerial  vehicle  designed  to  provide  a  ground  attack  capability 
during  reconnaissance  and  surveillance  missions.  Officials  stated  that  the 
MRL  assessment  results  have  (1)  identified  five  areas  that  needed  review 
prior  to  a  milestone  C  production  decision;  (2)  identified  two  risks  to  full- 
rate  production — mitigations  are  in  progress;  and  (3)  provided  evidence  to 
support  the  contractor’s  ability  to  meet  the  production  goal  of  two  aircraft 
per  month.  To  ensure  that  manufacturing  requirements  are  enforced, 
officials  have  developed  policy  for  programs  managers  to  assess 
manufacturing  readiness  at  key  decision  points.  To  support  that  policy,  the 
Air  Force  has  developed  training  for  integrated  product  teams  to  execute 
the  manufacturing  readiness  assessments.  Also  in  August  2009,  the  Air 
Force  Institute  of  Technology  established  a  Manufacturing  Readiness 
Assessment  course  to  provide  training  for  the  assessments  within  the  Air 
Force  and  is  currently  open  to  all  services  and  industry. 


DOD’s  Proposed 
MRLs  Embody  Many 
Best  Practices  of 
Leading  Commercial 
Firms 


To  successfully  develop  and  manufacture  their  products,  the  commercial 
firms  we  visited  used  a  disciplined,  gated  process  that  emphasized 
manufacturing  criteria  early  and  throughout  the  product’s  development. 
To  measure  manufacturing  maturity,  these  firms  developed  processes  that 
give  manufacturing  readiness  and  producibility  primary  importance 
throughout  the  product-development  process,  focusing  on  producing  a 
product,  not  developing  a  technology.  The  goal  is  business  profitability, 
and  manufacturing  maturity  is  important  to  this  process  from  the  earliest 
stages. 


The  best  practices  they  employed  were  focused  on  gathering  a  sufficient 
amount  of  knowledge  about  their  products’  producibility  in  order  to  lower 
manufacturing  risks  and  included  stringent  manufacturing  readiness 
criteria — to  measure  whether  the  product  was  mature  enough  to  move 
forward  in  its  development.  In  most  respects,  these  criteria  are  similar  to 
DOD’s  proposed  MRLs.  For  example,  as  with  MRLs,  commercial  firms 

•  assess  producibility  at  each  gate  using  clearly  defined  manufacturing 
readiness  criteria, 

•  gain  knowledge  about  manufacturing  early, 
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•  demonstrate  manufacturing  processes  in  a  production-relevant 
environment,  and 

•  emphasize  the  importance  of  effective  supply-chain  management. 

Essentially,  commercial  firms  emphasize  these  criteria  in  order  to 
maximize  their  understanding  of  manufacturing  issues,  to  mitigate 
manufacturing  risks  that  could  affect  business  profitability  or  schedule 
goals  for  getting  the  product  to  market.  DOD’s  MRLs  were  designed  to 
mitigate  similar  manufacturing  risks.  However,  the  difference  is  that  the 
commercial  firms  we  visited  required  that  their  manufacturing  processes 
be  in  control  prior  to  low-rate  production,  whereas  DOD’s  proposed  MRL 
criteria  do  not  require  as  early  control  of  the  manufacturing  process. 


DOD’s  MRLs  Are  Similar  to  Leading  commercial  firms  use  manufacturing  readiness  criteria,  similar  to 
Manufacturing  Criteria  DOD’s  MRLs,  to  assess  the  producibility  of  a  system,  gathering  knowledge 

Used  by  Leading  Firms  about  the  producibility  of  a  product  and  the  maturity  of  the  manufacturing 

^  process.  These  criteria  are  applied  early,  even  before  a  product  formally 

enters  into  development,  to  identify  and  manage  manufacturing  risks  and 
gaps.  Additional  manufacturing  readiness  criteria  are  applied  through  all 
the  stages  of  a  product’s  development  and  production  until  the  product  is 
ready  for  commercial  release.  The  firms  we  visited  used  manufacturing 
readiness  criteria  to  measure  both  the  readiness  of  the  product  or  material 
to  enter  into  development  and  to  proceed  through  the  necessary  gates. 
Table  3  below  shows  examples  of  manufacturing  readiness  criteria  that 
are  common  to  both  the  MRLs  and  the  commercial  criteria,  to  illustrate 
their  similarities.  Both  emphasized  identifying  risks  and  developing  plans 
to  mitigate  these  risks,  setting  realistic  cost  goals,  and  proving  out 
manufacturing  processes,  material,  and  products. 
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Table  3:  Many  Manufacturing  Criteria  Used  by  Leading  Commercial  Firms  Are  Similar  to  DOD’s  MRLs 


MRL  /  phases 
MRL  1-3 

Pre-Concept  Development 
(Invention  Stage) 

MRL  4 

Concept  Development 


MRL  5-6 

Technology  Development 


MRL  7 

Product  Development 


MRL  8 

Production  (Preparation) 


Commercial  manufacturing  criteria  and  DOD  MRLs 

•  Relevant  materials  and  processes  evaluated  for  manufacturability 

•  Cost  models  developed  for  new  processes 

•  Critical  manufacturing  processes  identified 

•  Risk-mitigation  plans  in  place  for  management  of  manufacturing  risks 

•  Key  materials  issues  identified 

•  Manufacturing  strategy  developed  and  integrated  with  acquisition  strategy 

•  Basic  design  requirements  defined  and  all  critical  technology  and  components  tested  and 
evaluated 

•  Critical  suppliers  identified  /  supply  chain  in  place 

•  Realistic  cost  targets  are  set 

•  Manufacturing  processes  and  materials  demonstrated  in  a  production-relevant  environment 

•  Product  requirements  and  features  well-defined 

•  P  ilot  lines'  yield-data  gathered  and  assessed 

•  Manufacturing  processes  demonstrated  in  a  production-representative  environment 

•  Quality  targets  demonstrated  on  pilot  line 

•  Manufacturing  processes  verified  for  low-rate  production  on  pilot  line 

•  Yield  and  rates  required  to  begin  low-rate  production  verified 

•  Manufacturing  plan  completed  and  all  key  manufacturing  risks  mitigated 

Source:  GAO  analysis  of  DOD  and  commercial  data. 


Best  Practice:  Commercial 
Companies  Emphasize 
Manufacturing  Criteria 
Early  and  at  Every  Stage  of 
the  Product-Development 
Life  Cycle 


Each  commercial  firm  we  visited  developed  a  disciplined  framework  for 
product  development  that  assessed  producibility  at  each  gate  using  clearly 
defined  manufacturing-maturity  criteria  that  are  similar  in  many  respects 
to  DOD’s  MRLs.  These  include  assessments  of  all  aspects  of 
manufacturing  technology  and  risk,  supply-chain  issues,  production 
facilities  and  tooling,  and  materials.  Throughout  the  product-development 
life  cycle,  these  criteria  were  applied  to  determine  entry  or  exit  into  the 
next  phase  and  led  to  informed  decisions  about  whether  the  product  was 
ready  to  move  forward  in  its  development.  Manufacturing  risks — such  as 
those  found  in  new  manufacturing  technologies  or  production  facilities, 
new  or  revolutionary  materials  or  supply-chain  issues — were  assessed  at 
each  step.  Deliverables,  including  risk-identification  and  mitigation  plans, 
manufacturing  plans,  and  funding  and  resource  needs,  were  required  at 
each  gate  in  order  to  progress  to  the  next  product-development  gate. 
Targets  were  developed  for  each  gate,  including  cost,  schedule,  and  yield 
goals,  and  the  product  team  was  responsible  for  either  meeting  these 
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targets  or  having  risk-mitigation  plans  in  place  if  the  targets  had  not  been 
met. 

GE  Aviation  exemplifies  this  disciplined  process,  using  a  highly  structured 
gated  process  with  detailed  checklists  for  entry  and  exit  into  each  phase. 
Like  DOD’s  MRLs,  these  checklists  contain  increasingly  detailed  criteria — 
as  they  move  from  product  start  to  production — for  evaluating 
manufacturing  technologies,  cost  drivers,  materials,  and  supply-chain 
issues.  Structured  teams  are  brought  together,  tools  are  identified  for 
execution  and  control  of  the  process,  and  scheduled  reviews  are 
conducted  with  defined  deliverables  and  checklists  for  each  milestone.  At 
each  milestone,  a  vigorous  review  of  the  plans  for  the  product’s 
development  and  manufacturing  and  risk-reduction  efforts  highlights 
issues  before  they  become  problems.  The  firm’s  goal  is  to  have  mature 
processes  by  production.  To  achieve  this,  it  considers  manufacturing 
readiness  throughout.  Each  project’s  team  is  cross-functional  and  includes 
senior  management,  mid-management  and  the  project  team.  This  robust 
review  process  leverages  expertise  across  GE  Aviation,  reduces  risk,  and 
highlights  issues  before  they  become  problems. 

As  with  all  the  commercial  firms  we  visited,  GE  Aviation  requires  strong 
management  involvement  at  each  gate,  along  with  decision  reviews  to 
determine  if  enough  knowledge  is  available  and  risk-mitigation  plans  are 
in  place  to  proceed  or  if  actions  to  address  and  mitigate  manufacturing 
risks  can  show  a  viable  way  forward.  This  allows  management  to  resolve 
problems  rather  than  pass  them  on  to  the  next  phase.  At  project  start, 
which  corresponds  to  MRL  4,  the  senior  leadership  team  and  product 
leadership  team  generate  the  product  idea  and  assess  the  need  for  the 
project.  They  provide  linkage  between  the  business  strategy  and  the 
project  and  develop  the  high-level  project  strategy.  They  identify  any  new 
product  material  or  manufacturing  processes  and  begin  to  develop  a  risk- 
reduction  strategy  for  these  issues.  By  the  time  the  product  enters  the 
preliminary  design  phase,  senior  leadership  and  project  teams  agree  on  the 
approach  to  the  project.  At  this  time,  product  directors  must  have  a 
manufacturing  plan  in  place  in  order  to  identify  how  they  are  going  to 
achieve  manufacturing  readiness.  Technical  risks  are  identified  in  the 
manufacturing  plan,  as  well  as  risk-abatement  strategies  for  materials  and 
manufacturing  processes  and  supply-chain  risks.  The  plan  has  to  show 
how  issues  will  be  successfully  addressed  by  the  detailed  design  phase, 
when  leadership,  the  project  team,  and  customers  agree  on  the  product  to 
be  delivered.  If  agreement  is  reached,  they  freeze  the  project  plan  and  a 
decision  is  made  to  fund  or  terminate  the  project. 
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Multidisciplinary  Team  / 
Manufacturing  Experts 


Maturing  Technology  and 
Manufacturing  Processes 


In  the  commercial  firms  we  visited,  product-development  teams  were 
multidisciplinary,  generally  including  management,  manufacturing,  quality, 
finance,  suppliers,  and  engineering,  with  necessary  skills  available  to 
assess  manufacturing  readiness.  Leading  firms  recognize  the  value  of 
having  a  knowledgeable,  well-trained,  and  skilled  manufacturing 
engineering  workforce  involved  in  these  multidisciplinary  teams  from  the 
beginning  and  throughout  the  process.  When  Honeywell  reorganized  its 
aerospace  business  in  2005,  it  created  an  advanced  manufacturing 
engineering  organization  to  focus  on  manufacturing  concerns  in  the 
earliest  phases  of  new  product-development  programs.  This  organization 
consists  of  engineers  to  support  various  manufacturing  disciplines  in 
Honeywell.  An  important  part  of  this  advanced  engineering  organization  is 
its  technology  group,  which  consists  of  a  select  number  of  technology 
fellows  with  extensive  expertise  in  key  manufacturing  disciplines  that 
touch  nearly  all  the  products  Honeywell  produces.  Honeywell  retains 
highly  skilled  manufacturing  expertise  through  this  program  and  uses 
these  experienced  and  knowledgeable  manufacturing  engineers  to  oversee 
each  project’s  manufacturing  assessments. 

Commercial  firms  focus  on  maturing  and  validating  technology  and 
manufacturing  processes  before  these  are  associated  with  a  product  and 
before  entry  into  the  gated  process.  They  keep  invention  and  unproven 
technologies  in  the  technology  base  until  their  producibility  at  the  scale 
needed  can  be  proven.  As  an  example,  GE  Healthcare’s  Gemstone 
scintillator  underwent  years  of  laboratory  development  on  a  small  scale 
until  GE  Healthcare  was  satisfied  that  this  material  was  ready  to  be  used 
on  its  computed  tomography  (CT)  scanners.  Scintillators  work  by 
converting  the  X-rays  in  the  CT  scanner  into  visible  light.  GE  Healthcare 
had  been  manufacturing  its  own  scintillators  since  the  late  1980s,  but  it 
needed  an  improved  one  that  worked  faster,  for  better  clarity  of  vision  and 
to  reduce  the  amount  of  exposure  to  radiation.  In  2001,  the  firm  began 
basic  composition  development  at  the  laboratory  scale  and  narrowed 
down  the  alternatives  to  find  the  material  with  the  best  properties  for  this 
use.  Even  at  this  early  stage,  several  years  before  the  material  would  enter 
into  GE  Healthcare’s  gated  process,  there  was  early  engagement  by  the 
chemists  with  the  manufacturing  side.  Before  they  decided  on  a  solution,  a 
determination  was  made  that  it  could  produce  them  with  sufficient  yield 
and  quality:  even  if  a  material  had  the  best  optical  qualities,  it  had  to 
balance  this  with  its  producibility.  GE  Healthcare  tested  thousands  of 
alternatives  to  determine  what  could  meet  its  technical  requirements  and 
be  producible  in  the  quantities  needed.  The  firm  narrowed  it  down  to  a 
garnet-based,  rare-earth  minerals  composite,  and  began  producing  it  in 
small  but  increasing  quantities.  After  narrowing  the  field  to  this  gamet- 
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based  compound,  GE  Healthcare  began  to  determine  its  suppliers  and 
what  equipment  was  needed.  The  firm  then  began  building  its  first  pilot 
plant  to  produce  the  material  and  the  scintillators,  2  years  before  the 
scintillator  entered  the  firm’s  gated  process.  Figure  4  shows  a  photo  of  a 
CT  scanner  that  uses  the  scintillator  technology. 


Figure  4:  GE  CT  Scanner  Using  Advanced  Scintillator  Material 


Source:  Copyright  ©  General  Electric  Company  USA.  All  rights  reserved. 


Best  Practice:  Commercial 
Firms  Have  Adopted 
DOD’s  MRLs  or  Are 
Employing  Similar  Criteria 
in  Their  Product- 
Development  Process 


Because  leading  commercial  firms  focus  on  producibility  as  a  key  element 
to  successfully  develop  products,  they  use  rigorous  analysis  methods  to 
assess  producibility  and  to  identify  and  manage  manufacturing  risks  and 
gaps.  They  apply  these  methods  and  tools  early  and  throughout  product 
development  and  use  them  to  manage  their  product  development  on  a 
daily  basis.  This  commercial  approach  is  a  process  in  which  quality  is 
designed  into  a  product  and  manufacturing  processes  are  brought  into 
statistical  control  to  reduce  defects,  in  contrast  to  practices  employed  by 
many  defense  contractors  where  problems  are  identified  and  corrected 
after  a  product  is  produced. 
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Some  firms  were  familiar  with  the  DOD  MRL  proposal  and  had  taken  steps 
to  use  the  concepts  at  their  own  companies.  Honeywell,  for  example, 
determined  that  early  decisions  were  responsible  for  many  production 
issues  and  so  they  developed  analytical  tools  and  models  that  support 
evaluations  of  manufacturing  and  risk  throughout  the  product- 
development  life  cycle.  In  2005,  Honeywell  engineers  began  looking  for  a 
way  to  measure  manufacturing  readiness  and  producibility,  since  they 
realized  that  early  program  decisions  were  driving  many  production  issues 
and  that  by  the  time  a  product  entered  engineering  and  manufacturing 
development,  it  was  too  late  to  efficiently  affect  these  issues.  Some  of 
these  issues  include  cost  overruns,  quality  problems,  low-yield  issues, 
service  and  maintainability  inefficiencies,  and  supply-chain  problems. 

A  literature  search  led  them  to  DOD’s  MRLs  and  they  realized  that  these 
could  provide  the  type  of  metric  needed  for  a  quantitative  assessment. 
Honeywell  then  evolved  its  own  criteria  from  these  MRLs,  modified  to 
meet  Honeywell’s  needs  and  expanded  to  address  concerns  such  as 
design,  obsolescence,  and  testability  issues.  Their  MRL  Maturity  Model 
assessment  tool,  which  evolved  from  an  early  version  of  DOD’s  MRLs,  is 
the  main  tool  in  the  assessment  and  is  built  upon  three  enabling 
producibility  analysis  tools.  The  model  provides  an  MRL  score  for  the 
product  “as  is,”  which  is  then  compared  to  the  MRL  score  desired  to  exit 
the  phase.  This  model  gives  the  firm  a  systematic  way  to  be  sure  all  the 
information  is  considered  and  the  right  questions  are  asked  by  less- 
experienced  engineers  who  support  the  program.  This  MRL  tool  was 
developed  5  years  ago  and  has  evolved  in  an  iterative,  continuous 
improvement  process  since  then,  based  on  feedback  from  its  users.  Figure 
5  shows  a  simplified  depiction  of  this  MRL  model  and  the  three  enabling 
tools. 
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Figure  5:  Honeywell  Uses  Three  Producibility  Models  and  MRL  Workshop 


Manufacturing 
Complexity  Model 

•  Identifies  design  attributes 
driving  manufacturing 
complexity 


Yield  Prediction  Model 

•  Quantifies  anticipated  yield 
of  proposed  design 
concepts 


MRL  Assessment  Tool 

B  uilt  on  inputs  from  three 
producibility  analysis 
tools  and  MRL  workshop  to 
evaluate  manufacturing 
maturity  and  identify  gaps 


Design  for 
Manufacturing 
Scorecard  Analysis 

•  Quantifies  the  impact  of 
design  for  manufacturing 
violations 


MRL  Workshop 

•  Review  manufacturing 
maturity  artifacts  against 
evaluation  criteria  to  gain 
concensus  on  ratings  for 
each  category 


MRL 

Maturity 

Model 

Output: 

■MRL  score 


Source:  GAO  analysis  of  Honeywell  data. 


The  output  of  this  tool  is  an  MRL  assessment  score  that  can  identify  gaps 
or  risks.  For  example,  spreadsheets  show  the  MRL  scoring  at  a  glance  for 
each  of  the  elements  evaluated,  pinpointing  the  gaps;  risk  worksheets  to 
quantify  the  risks;  and  action  plans  to  close  the  gaps  and  mitigate  these 
risks.  It  links  to  the  firm’s  gated  process,  providing  entry  and  exit  criteria 
and  feedback  on  how  to  meet  these  criteria.  The  important  information 
obtained  is  not  necessarily  what  MRL  level  the  item  is  at  currently,  but 
rather  the  robustness  of  the  gap-closure  plan  to  get  to  the  desired  level  for 
the  next  gate.  The  application  of  the  MRL  tool  helps  identify  what  these 
key  gaps  are  and  what  steps  are  required  to  close  them. 
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The  three  enabling  producibility  tools  that  provide  support  for  this 
assessment  and  early  input  on  the  producibility  risks  are:  a  Design  for 
Manufacturing  Model,  a  Product  Complexity  Model  and  a  Yield  Prediction 
Model: 

•  Manufacturing  Complexity  Model:  This  model  identifies  the  design 
features  that  are  driving  manufacturing  complexity  into  the  design  and 
enables  scenarios  to  be  evaluated  to  see  what  actions  can  be  taken  to 
simplify  the  design.  Higher-complexity  designs  generally  cost  more  and 
are  higher  risk,  so  the  goal  is  to  identify  alternative  design  solutions 
that  minimize  complexity,  but  still  meet  all  the  performance 
requirements. 

•  Yield  Prediction  Model:  Honeywell  has  also  developed  yield  prediction 
models  based  on  statistical  principles  that  correlate  opportunities  for 
defects  in  a  design  to  established  process  capability  benchmarks.  This 
approach  is  used  to  predict  yield  during  early  design  activities  based 
on  knowledge  of  the  manufacturing  processes  used  and  the  complexity 
of  the  design. 

•  Design  for  Manufacturing  Scorecard  analysis:  The  third  Honeywell- 
developed  tool  is  a  design  for  manufacturing  scorecard,  which 
quantifies  how  well  the  design  adheres  to  recommended  best 
practices.  The  goal  of  using  the  tool  is  to  provide  feedback  to  the 
designers  so  that  they  see  how  their  design  decisions  directly  affect 
producibility  and  help  pinpoint  improvement  areas  early  in  the 
process. 

Honeywell  then  conducts  an  MRL  workshop,  with  a  team  led  by  an 
engineer  from  its  Advanced  Manufacturing  Engineering  group  that 
includes  the  program  manager  and  various  subject-matter  experts.  This 
team  reviews  the  tools  and  the  MRL  criteria  to  gain  consensus  on  ratings 
for  each  category.  Honeywell’s  Manufacturing  Maturity  Model,  with  input 
from  these  enabling  tools,  is  used  to  develop  an  MRL  score  for  the 
product.  These  assessments  provide  early  producibility  evaluations 
essential  to  mitigating  design-driven  risks.  Since  many  producibility  issues 
are  driven  by  early  design  architecture  decisions,  these  tools  provide  a 
way  to  analyze  these  decisions  early  and  make  the  necessary  performance 
and  producibility  trades  through  “virtual  prototyping”  long  before  actual 
hardware  is  built.  The  MRL  score  provides  the  necessary  framework  to  ask 
the  questions  that  such  an  analysis  needs  to  answer. 

After  the  MRL  assessment  is  complete  and  the  MRL  scores  and  risk- 
mitigation  plans  are  approved,  the  MRL  analysis  and  risk  mitigations  are 
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incorporated  into  the  daily  schedule  of  the  program  office.  The  office 
continually  monitors  the  MRL  levels,  updating  them  and  working  toward 
its  risk  mitigation  goals. 

Best  Practice:  Leading 
Firms  Prove  Out 
Manufacturing  Tooling, 

Equipment,  and  Processes 
before  Entry  into 
Production 

GE  Aviation  officials  told  us  that  certain  advanced  manufacturing 
technologies  achieve  significant  cost  savings  by  getting  the  costs  lower 
earlier  in  the  process  and  decreasing  cycle  time  for  faster  implementation. 
An  example  of  manufacturing  techniques  or  processes  that  have  made  a 
big  difference  in  costs,  accuracy,  and  reliability  include  processes  for 
drilling  small  shaped  holes  for  turbine  airfoils. 

GE  Aviation’s  Turbine  Airfoils  Lean  Lab  provides  a  mock-up  of  a 
production  facility  or  process,  where  such  technologies  and  production 
processes  can  be  tested  to  eliminate  waste,  scrap,  and  excess  steps.  They 
focus  on  one  part  family  or  process,  such  as  the  turbine  airfoil  shaped-hole 
manufacturing.  The  turbine  airfoil  is  a  part  of  the  jet  engine  that  generates 
power — it  extracts  horsepower  from  the  high-temperature,  high-speed 
combusted  gasses.  Turbine  airfoil  blades  require  hundreds  of  cooling 
holes  that  help  maintain  part  integrity  at  elevated  operating  temperatures. 
Traditionally,  round  holes  were  used,  but  the  technology  has  evolved  to 
compound-angle-shaped  holes,  which  improve  cooling  effectiveness  and 
reduce  engine  stress.  These  type  of  holes  cannot  be  economically 
produced  by  traditional  methods  and  require  improved  manufacturing 
techniques.  Advanced  laser  drilling  was  determined  to  be  feasible,  and  GE 
Aviation  decided  to  initiate  the  program  through  the  Lean  Lab  to  ensure 
manufacturing  readiness  of  the  process. 

GE  Aviation  officials  compared  their  processes  in  this  case  to  DOD’s 
MRLs.  Prior  to  entering  their  gated  process,  they  began  making 
investments  in  potential  technologies,  including  tooling  (MRL  1-3).  As  the 
gated  process  began,  risks  were  identified  and  risk-abatement  plans  were 
put  in  place  (MRL  4).  GE  Aviation  then  set  up  the  Lean  Lab  to  test  the  way 
the  airfoil  would  actually  be  built.  New  processes  were  introduced  that 
included  new  laser  methods  for  hole  drilling,  improved  robotic  technology, 
machining,  and  grinding  (MRL  5-6).  The  managers  then  ran  the  pilot 
production  line  for  some  time  to  manufacture  these  airfoils  using  actual 


Companies  we  visited  spent  years  prior  to  production  developing  and 
proving  out  their  manufacturing  processes,  including  building  test  articles 
on  pilot  production  facilities  to  perfect  these  processes.  This  allowed  them 
to  perfect  and  validate  these  processes,  eliminate  waste  and  scale  up 
gradually  to  the  required  manufacturing  level.  They  reduce  errors  and 
inefficiencies  with  the  purpose  of  retiring  manufacturing  risks. 
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production  operators  to  be  confident  that  the  process  would  translate  to 
the  production  line.  Adjustments  were  made  to  improve  efficiency  and 
retested  on  the  line  until  they  were  satisfied  that  they  had  worked  out  the 
best  procedures.  GE  had  tooling-design  experts  on  the  team  at  the  Lean 
Lab  to  provide  rapid  part  and  tool  manufacturing.  Processes  were  brought 
into  statistical  control  in  order  to  take  the  complexity  out  of 
manufacturing,  simplify  the  process,  and  reduce  waste  (MRL  7-8).  They 
then  dismantled  the  production  line  at  the  Lean  Lab,  took  it  to  the 
manufacturing  facility,  and  set  it  up  exactly  the  same,  with  no  variations 
allowed  (MRL  9).  This  seamless  introduction  of  the  new  manufacturing 
technology  and  the  lean  principles  developed  in  the  lab  are  expected  to 
save  many  millions  of  dollars  across  GE  Aviation,  on  production  of  this 
part  family  alone.  Figure  6  shows  a  photo  of  GE  Aviation’s  Lean  Lab  setup. 


Figure  6:  GE  Aviation’s  Turbine  Airfoils  Lean  Lab  Proves  Out  Production  Processes 


Source:  Copyright  ©  General  Electric  Company  USA.  All  rights  reserved. 

GE  Aviation's  Turbine  Airfoils  Lean  Lab  provided  a  seamless  way  to  introduce  new  manufacturing 
processes. 


GE  Healthcare  provides  another  example  of  proving  out  manufacturing 
processes  prior  to  production  in  their  development  of  the  Gemstone 
scintillator  for  use  on  their  CT  Scanners.  In  2003,  the  technology  for  this 
transitioned  into  the  firm’s  formal  gated  process  or  product  start-up,  and  it 
began  a  detailed  and  extensive  development  of  the  manufacturing  process. 
The  firm  built  a  pilot  plant  for  this  purpose  and  began  manufacturing  the 
composite  in  increasing  amounts.  In  this  first  pilot  plant,  it  was  able  to 
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process  the  materials  in  increased  quantities  from  what  it  produced  in  the 
lab.  GE  Healthcare  verified  that  it  had  the  right  technologies  to  minimize 
manufacturing  risks.  In  the  laboratory  environment,  the  firm  had  already 
answered  the  question  “Can  this  composite  be  made  with  the  desired 
properties?”  and  now  asked  “Can  it  be  made  with  sufficient  yield  and 
quality  to  be  manufactured  in  the  desired  amounts?”  This  early 
engagement  with  manufacturing  enabled  the  firm  to  develop  the  process 
and  reduce  errors  and  inefficiencies  with  the  purpose  of  reducing 
manufacturing  risks. 

GE  Healthcare  then  built  a  second  pilot  production  plant  that  further 
increased  the  amount  produced  above  that  of  the  first  pilot  plant.  The  firm 
continued  its  focus  on  gaining  knowledge  early,  but  on  a  larger  scale: 
building  the  pilot  plants  was  important  to  perfecting  the  process  and 
gaining  knowledge  about  the  material’s  producibility.  At  this  stage,  which 
coincides  with  MRL  8,  it  eliminated  most  of  the  technical  risks  involved  in 
manufacturing  the  material.  The  firm  then  began  to  build  its  full-scale 
facility,  which  was  ready  18  months  before  product  launch. 


When  the  full-scale  production  facility  was  completed,  further  scale-up  of 
the  material’s  manufacturing  became  the  focus.  Changes  to  the  design 
were  made  as  needed  to  facilitate  this.  Any  remaining  manufacturing  risks 
were  eliminated  prior  to  entry  into  the  next  stage,  the  product-validation 
stage.  The  Food  and  Drug  Administration  requires  validation  of  finished 
medical  devices.  GE  Healthcare  told  us  that  this  means  that  all  the 
equipment,  processes,  procedures,  and  factory  workers  are  the  same  as 
will  be  used  in  actual  production.  Through  use  of  the  pilot  plants  to 
perfect  the  manufacturing  of  the  scintillator  material,  GE  Healthcare  was 
able  to  produce  production-representative  material  to  satisfy  this 
requirement. 


Best  Practice:  Commercial 
Firms  Work  Closely  with 
Suppliers,  Who  Must  Meet 
High  Quality  Standards  for 
Parts  and  Supplies 


Commercial  firms  focus  on  developing  strong  relationships  with  their 
suppliers  to  ensure  quality  parts  are  provided  in  a  timely  manner.  This 
begins  with  rigorous  supplier-selection  criteria  to  create  a  strong  supplier 
base  to  provide  quality  parts.  Similarly,  DOD’s  MRL  supply-chain  thread 
focuses  on  supplier  capability  throughout  the  acquisition  life  cycle,  from 
as  early  as  pre-milestone  A  (MRL  3),  where  initial  assessment  of  the 
supply  chain  begins,  through  MRL  5,  where  supply-chain  sources  have 
been  identified,  and  continuing  to  MRL  8,  where  the  supply-chain  should 
be  stable  and  adequate  to  support  low-rate  production.  Commercial  firms 
generally  have  long-term  relationships  with  these  suppliers  and  can 
identify  the  supplier  that  is  the  best  source  of  material  or  parts  early,  well 
before  production  begins.  Leading  commercial  firms  apply  the  same 
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standards  to  these  suppliers  as  they  apply  to  their  own  manufacturing 
processes,  such  as  ISO  900021  or  other  quality  standards.  Throughout 
product  development  and  production,  they  establish  effective 
communications  with  their  suppliers  so  they  can  continually  assess  their 
performance.  These  firms  work  closely  with  their  suppliers  to  retain  these 
beneficial  relationships,  providing  training  where  necessary  and  assistance 
if  manufacturing  problems  arise. 

GE  Healthcare  suppliers  have  to  be  validated  before  production  begins, 
but  qualifying  them  starts  in  the  design  phase.  Suppliers  are  expected  to 
meet  the  ISO  9000  standards  and  the  Food  and  Drug  Administration’s 
medical  devices  standards,  but  GE  Healthcare’s  own  standards  are  more 
stringent  that  those.  The  supplier-qualification  process  ensures  that 
suppliers  meet  GE  Healthcare’s  requirements,  have  a  quality  system  that 
provides  the  appropriate  controls  for  the  part  provided  and  meet 
regulations  and  requirements  of  multiple  agencies,  such  as  the  Food  and 
Drug  Administration.  Once  a  supplier  is  qualified,  it  becomes  an  approved 
supplier. 

GE  Healthcare  also  audits  most  of  its  suppliers  and  looks  for  issues  such 
as  lapsed  ISO  9000  certification  or  a  failed  review.  If  it  finds  these  things, 
GE  Healthcare  will  ask  the  supplier  for  a  plan  to  correct  the  deficiency  and 
reaudit  the  supplier.  GE  Healthcare  does  annual  risk  assessments  on  the 
suppliers,  based  on  data  gathered  during  these  audits,  with  sole-source  or 
single-source  suppliers  being  a  high  risk.  If  a  supplier  falls  out  of  qualified 
status,  GE  Healthcare  will  do  more  frequent  assessments.  It  constantly 
monitors  the  suppliers  for  quality.  It  helps  the  supplier  get  to  the  quality 
needed,  but  quality  goals  must  be  met. 

Siemens  is  a  global  company  that  employs  about  70,000  people  in  the 
United  States.  We  visited  Siemens  Mobility  Division,  which  builds  light  rail 
cars  for  public  transit.  Siemens  places  special  emphasis  on  its  supplier 
relationships,  since  it  knows  its  suppliers  can  contract  to  other  rail-car 
builders,  as  there  is  competition  for  suppliers  in  this  market.  If  it  has  a 
good  relationship  with  its  suppliers,  it  can  continue  to  benefit  from  the 
relationships  with  high-quality  suppliers.  Once  it  qualifies  a  supplier,  it 
takes  the  responsibility  for  keeping  the  supplier  qualified,  providing 


JIAn  ISO  certification  against  a  standard,  such  as  ISO  9000,  means  that  an  independent 
external  body  has  audited  an  organization’s  quality-management  system  and  verified  that  it 
conforms  to  the  requirements  specified  in  the  standard. 
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technical  assistance  if  necessary  to  keep  the  supplier  in  its  pipeline.  Even 
as  early  as  the  bid  phase  of  the  contract,  Siemens  knows  who  it  will  need 
as  suppliers  and  if  any  particular  supplier  is  new  or  challenged  in  some 
respect. 


Siemens  applies  a  three-step  supplier-qualification  process  to  its  suppliers. 
This  starts  with  a  supplier  self-assessment.  The  firm’s  supplier- 
qualification  personnel  then  visit  the  supplier’s  plant  and  evaluate  the 
supplier  on  the  same  self-assessment  form,  to  determine  if  the  supplier 
will  make  it  to  the  vendor-qualification  list.  Once  a  supplier  is  on  the 
approved  vendor-qualification  list,  Siemens  does  risk  ratings  for  these 
vendors  to  be  sure  it  can  keep  them  on  the  qualified-vendor  list.  The  firm 
updates  these  assessments  if  the  vendor  situation  changes,  rating  the 
vendor  at  low  risk  if  it  is  fully  qualified  and  working  with  it  if  some  aspects 
are  not  qualified.  Siemens  takes  responsibility  for  keeping  the  approved 
suppliers  qualified,  since  finding  and  qualifying  new  vendors  can  be  time- 
consuming  and  risky.  It  tries  not  to  overload  any  one  supplier,  because 
some  of  their  suppliers  are  small  or  specialty  operations,  so  it  keeps  a  pool 
of  qualified  suppliers  for  as  many  parts  or  materials  as  it  can. 


Commercial  Firms  Require 
That  Manufacturing 
Processes  Be  in  Control 
Earlier  Than  DOD’s  MRLs 


Although  the  firms  we  visited  used  manufacturing  readiness  criteria 
similar  to  DOD's  proposed  MRLs,  one  important  difference  we  observed  is 
that  the  commercial  best  practice  is  to  have  manufacturing  processes  in 
control  prior  to  the  production  decision,  while  DOD's  MRLs  require 
manufacturing  processes  and  procedures  to  be  established  and  controlled 
during  MRL  9,  which  occurs  after  the  milestone  C  production  decision, 
which  authorizes  a  program  to  enter  low-rate  initial  production,  or 
equivalent.22  Although  DOD’s  MRLs  incorporate  many  of  the  commercial 
manufacturing  best  practices  into  their  manufacturing  design  and 
implementation  criteria,  the  process  controls  criteria  would  be  met  too 
late  in  the  process  to  achieve  their  full  effect.  DOD’s  MRL  matrix  states 
that  low-rate  production  yield  and  rate  targets  should  be  achieved  at  MRL 
9,  after  the  production  decision  has  been  made.  The  commercial  firms  we 
talked  to  emphasized  that  production  processes  must  be  in  control  before 
this  decision  is  made.  They  realize  that  they  are  unable  to  make 
predictions  about  production  performance  until  the  process  is  stable  and 
defects  are  predictable.  Not  achieving  process  control  could  result  in  low 
quality,  extensive  rework  and  waste,  and  not  meeting  cost  and  schedule 


22Process  controls  are  the  use  of  statistical  methods  to  monitor,  track,  and  reduce  the 
variability  of  manufacturing  processes. 
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targets.  Firms  established  pilot  lines  to  prove  out  production  material, 
processes,  and  tooling,  and  worked  to  get  processes  under  control  before 
the  system  could  move  from  the  pilot  line  to  production  line.  Figure  7 
shows  a  depiction  of  the  commercial  manufacturing  process  approach. 


Figure  7:  Leading  Commercial  Firms  Use  Statistical  Controls  to  Ensure  Quality  Products 
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Source:  GAO  analysis  of  commercial  firm  data. 


The  companies  we  visited  used  various  approaches  to  build  process 
capability  and  provide  timely  information  on  whether  manufactured 
components,  subsystems,  or  systems  meet  design  specification.  For 
example,  GE  Aviation  uses  a  statistical  measurement,  called  Z  sigma  level, 
to  determine  whether  its  processes  have  been  brought  under  control  or  if 
variations  in  its  manufacturing  process  could  affect  the  quality  of  the 
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Two  Successful  DOD 
Programs  Used  Criteria 
Similar  to  Commercial 
Firms 


product.  The  product  is  not  moved  into  production  until  the  firm  is 
satisfied  that  these  processes  are  in  control.  Similarly,  GE  Healthcare’s 
milestone  process  requires  that  a  set  of  quality  targets  are  part  of  the 
program  and  that  those  quality  targets  are  met.  Measures  of  process 
control  vary  from  company  to  company,  such  as  using  yield  or  scrap  and 
rework  rates  or  sigma  levels,  but  each  looks  carefully  at  those  measures  to 
ensure  they  carried  no  product-quality  risk  and  uses  this  information  to 
determine  if  the  product  is  ready  to  be  manufactured. 

Two  DOD  programs,  the  Army’s  Lakota  aircraft  and  the  Missile  Defense 
Agency’s  Standard  Missile  3  Block  1A,  that  had  successful  manufacturing 
outcomes  employed  some  of  the  same  practices  as  leading  commercial 
firms.  Both  used  a  type  of  manufacturing  readiness  criteria  to  evaluate 
whether  the  programs  were  ready  to  enter  into  production  and  both 
programs  focused  on  manufacturability  as  a  key  indicator  of  program 
success,  using  well-developed  technology  and  a  conservative  approach  in 
design  and  development. 

The  Lakota  aircraft,  a  light  utility  helicopter  that  conducts  noncombat 
missions,  was  a  mature  aircraft  design  when  the  Army  entered  into  the 
contract  with  the  European  Aeronautic  Defence  and  Space  Company  to 
purchase  this  commercially  available  helicopter.  The  program  shows  how 
careful  attention  to  manufacturing  readiness  can  reduce  program  risks. 
According  to  program  office  officials,  the  contractor  was  chosen  in  part 
because  of  its  manufacturing  track  record,  and  it  completed  extensive 
planning,  both  internally  and  with  its  supplier  base,  to  ensure  on-time  and 
reliable  deliveries.  Production  planning  and  preparation  were 
accomplished,  including  assessments  of  the  manufacturing  processes, 
capabilities,  and  facilities.  These  assessments  determined  that  the  program 
was  low  risk  and  ready  for  full-rate  production.  The  Lakota  is  currently  in 
full-rate  production  and  has  met  its  cost  and  schedule  targets. 

The  Standard  Missile  3  is  a  ship-based,  antiballistic  missile  used  by  the 
Aegis  ballistic  missile  defense  system.  Similar  to  the  Lakota,  the  system 
met  its  cost  and  schedule  goals  by  using  an  incremental,  low-risk 
approach.  Like  the  commercial  firms  we  visited,  the  program  built 
knowledge  through  the  use  of  a  type  of  manufacturing  readiness  criteria, 
which  allowed  the  early  identification  of  risk  and  implementation  of 
mitigation  strategies.  The  Standard  Missile  3  Block  LA  was  also  on  target 
for  manufacturing  cost  and  schedule  and  reported  a  lower  cost  per  unit 
than  was  originally  estimated  on  its  production  buys.  As  in  the  successful 
commercial  firms  we  visited,  manufacturing  issues  were  considered  very 
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MRLs  Are  Hampered 
by  Lack  of  an 
Agencywide  Policy 
and  Manufacturing 
Workforce  Concerns 


Draft  Policy  to 
Institutionalize  MRLs  Has 
Proven  Difficult,  but  the 
DOD  Community  Is 
Starting  to  See  Its  Value 


early  in  the  design  phase,  leading  to  minimal  changes  in  the  program  from 
flight  test  to  production. 


While  acceptance  of  MRLs  is  growing  within  DOD  and  the  defense 
industry,  the  services’  leadership  appears  to  be  resistant,  and  adoption 
efforts  have  been  slow.  For  example,  obtaining  agreement  on  a  policy  that 
would  institutionalize  MRLs  defensewide  has  proven  difficult.  Concerns 
raised  by  the  military-service  policymakers  have  centered  on  when  and 
how  the  MRL  assessments  would  be  used.  Officials  responsible  for  the 
draft  policy  have  promoted  MRLs  as  an  initiative  that  can  address  the 
manufacturing  element  in  the  design  and  production  of  weapon  systems, 
citing  commercial  best  practices  that  employ  similar  methods,  and 
benefits  derived  from  pilot  programs.  While  extensive  efforts  have  been 
made  to  promote  the  benefits  of  MRLs  in  support  of  a  revised  draft  policy, 
it  has  taken  nearly  2  years  to  allay  concerns  and  it  has  not  yet  been 
approved.  DOD  is  likely  to  face  serious  challenges  even  if  an  agreement  is 
reached  to  approve  the  policy,  however,  because  the  number  of  DOD’s 
production  and  manufacturing  career-field  employees  has  diminished, 
particularly  within  the  Air  Force.  Although  the  services  are  at  the 
beginning  stages  of  revitalizing  their  production  and  manufacturing 
workforce,  DOD  currently  does  not  have  adequate  in-house  expertise  with 
the  requisite  knowledge  to  assess  manufacturing  throughout  DOD. 
Essentially,  the  military  services  and  Defense  Contract  Management 
Agency  have  identified  knowledge  and  manpower  gaps  in  their 
manufacturing  workforce  and  believe  that  any  initiative  deploying  MRLs 
defensewide  could  be  hampered  as  a  result. 


While  acceptance  of  MRLs  is  growing  within  DOD  and  the  defense 
industry,  the  Army’s,  Navy’s,  and  Air  Force’s  leadership  appears  to  be 
resistant  and  adoption  efforts  have  been  slow.  For  example,  a  July  2008 
draft  MRL  policy  memorandum  garnered  disagreement  among  the  military- 
service  policymakers.  The  military  services’  leadership  agreed  that  MRLs 
provide  value  in  the  early  acquisition  phases  but  disagreed  with  the 
policy’s  intent  to  formalize  the  process.  For  example,  the  MRL  policy 
memorandum  stated  that  on  the  basis  of  analyses  by  GAO  and  the  Defense 
Science  Board — as  well  as  positive  results  on  two  Air  Force  pilot 
programs — that  acquisition  category  I  programs  be  assessed  using  the 
MRL  scale.  In  particular,  the  draft  policy  included  provisions  that  would 
require  programs  at  milestone  B  to  be  assessed  at  MRL  6  or  beyond  for  all 
critical  technologies;  programs  at  milestone  C  to  be  assessed  at  MRL  8  for 
all  critical  technologies;  procedures  to  be  coordinated  for  including 
assessments  of  manufacturing  readiness  in  addition  to  technology 
readiness  assessments  at  milestone  B  and  C;  and  incorporation  of 
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guidance  into  training  materials  and  guidebooks  on  best  practices  for 
addressing  manufacturing  from  the  earliest  stages  of  development  through 
production  and  sustainment. 

In  response  to  the  draft  policy,  each  of  the  military  services  issued 
memorandums  in  July  2008  to  the  Under  Secretary  of  Defense 
(Acquisition,  Technology  and  Logistics)  or  the  Director,  Defense  Research 
and  Engineering,  stating  they  support  MRLs  and  their  use  earlier  in  the 
acquisition  process  but  they  saw  limited  value  in  doing  formal  assessments 
prior  to  milestone  C.  In  general,  the  services  had  concerns  on  when  and 
how  MRL  assessments  would  be  used.  More  specifically,  their  concerns 
included  the  following: 

•  Evaluation  results  that  could  be  used  as  the  basis  for  go  /  no  go 
decisions. 

•  A  growing  number  of  assessments  being  levied  on  acquisition 
programs. 

•  Resources  required  to  prove  out  multiple  production  lines  in  a 
competitive  prototyping  environment  during  the  technology- 
development  phase. 


Since  2008,  officials  responsible  for  the  draft  policy  memorandum  have 
been  working  to  address  concerns  raised  by  the  services.  According  to  the 
working  group,  most  concerns  pointed  to  a  need  to  clarify  how  the 
information  is  intended  to  be  used  by  decision  makers  at  key  milestones, 
particularly  at  the  earlier  milestones.  According  to  the  working  group 
officials  we  interviewed,  the  intent  is  to  inform  decision  makers  with 
critical  information — such  as  manufacturing  risk  and  readiness  measures, 
as  appropriate  to  the  phase  of  acquisition — so  that  knowledge-based 
decisions  can  be  made  earlier  in  the  process  to  influence  better  outcomes 
in  terms  of  cost  and  schedule  in  the  later  acquisition  phases.  Moreover, 
they  cite  that  similar  methods  are  employed  by  leading  commercial  firms 
as  a  best  practice,  plus  the  fact  that  MRL  pilot  programs  have  already 
demonstrated  significant  benefits.  The  revised  MRL  draft  policy  has  not 
yet  been  approved.  Officials  familiar  with  the  status  of  the  draft  policy 
stated  that  the  leadership  at  one  of  the  military  services  is  still  opposed  to 
the  idea  of  standardizing  MRLs  across  DOD,  and  efforts  to  get  approval 
have  not  yet  occurred  within  the  Office  of  the  Director,  Defense  Research 
and  Engineering. 
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DOD  experienced  similar  problems  introducing  technology  readiness 
levels.  There  was  opposition  to  the  use  of  technology  readiness  levels,  but 
they  became  a  standard  for  programs  to  follow,  and  the  standard  that 
technologies  should  be  demonstrated  in  a  relevant  environment  became  a 
statutory  requirement  for  all  major  acquisition  programs  seeking  to  enter 
system  development.23  Programs  report  benefits  from  using  technology 
readiness  levels. 


Some  officials  believe  that  MRLs  could  significantly  reduce  cost  growth. 
For  example,  the  Army  and  Air  Force  have  reported  MRLs  were  a  factor 
that  contributed  to  benefits  of  hundreds  of  millions  of  dollars  in  reduced 
program  costs,  improved  schedule,  and  better  performance  of  products. 


MRL  Acceptance  Is 
Growing  within  DOD  and 
Defense  Industry 


A  number  of  Army,  Air  Force,  and  Missile  Defense  Agency  programs — as 
well  as  defense  contractors — have  embraced  MRLs  as  the  method  for 
assessing  manufacturing  maturity,  risk,  and  readiness.  For  example,  some 
Army  commands  have  opted  to  use  them  on  their  science  and  technology 
efforts  that  have  manufacturing  elements,  and  have  developed  a  formal 
process  for  identifying  them.  Similarly,  two  of  three  Air  Force  product 
centers  under  the  materiel  command — the  Aeronautical  Systems  Center 
and  the  Air  Armament  Center — have  recently  issued  local  policy  that 
mandate  the  use  of  MRLs.  For  example,  in  a  policy  memorandum  by  the 
Aeronautical  Systems  Center,  dated  October  13,  2009,  all  programs  are 
now  required  to  have  manufacturing  readiness  assessments  using  MRLs, 
prior  to  each  major  milestone  review.  The  memorandum  acknowledged 
that  the  transition  to  production  has  historically  been  challenging  for 
many  programs  and  that  manufacturing  assessments  are  a  key  tool  to 
ensure  that  programs  are  ready  to  begin  production.  The  Missile  Defense 
Agency  has  included  MRLs  as  part  of  their  assessment  criteria.  In  addition, 
senior  missile  defense  manufacturing  personnel  have  developed  and 
conducted  training  on  how  to  conduct  these  assessments. 


Similarly,  a  number  of  defense  contractors  have  implemented  MRLs  as  a 
discipline  for  identifying,  managing,  and  communicating  manufacturing 
risk  and  readiness.  These  contractors  report  a  number  of  benefits  using 
the  MRLs,  including  reductions  in  program  costs  and  improved  production 
schedule.  For  example,  in  2006,  Raytheon  participated  in  pilot  MRL 
program  assessments  involving  the  Advanced  Medium-Range  Air-to-Air 


23National  Defense  Authorization  Act  for  Fiscal  Year  2006,  Pub.  L.  No.  109-163,  §  801, 
codified  at  10  U.S.C.  §  2366b. 
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Missile  and  a  portfolio  of  other  programs  and  concluded  the  approach 
makes  good  business  sense  to  lower  risk.  Raytheon  claimed  cost 
reductions  of  30  percent  or  more  could  be  achieved  by  using  MRLs. 
Raytheon  officials  state  that  the  combination  of  technology  and 
manufacturing  assessment  processes  changes  the  culture  by  driving  a 
collaborative  partnership  between  programs,  design,  and  manufacturing 
engineering  earlier  in  the  product-development  life  cycle  where  maturity 
efforts  can  have  the  greatest  effect  on  improving  program  affordability  and 
predictability.  As  a  result,  Raytheon  is  deploying  MRLs  as  a  standard 
across  the  organization.  Lockheed  Martin  is  exploring  ways  to  integrate 
MRLs  within  its  existing  review  processes.  As  previously  discussed, 
Honeywell  adopted  MRLs  for  use  on  both  its  defense  and  commercial 
products,  and  developed  several  models  as  an  analysis-based  approach  to 
quantify  their  producibility  risks. 


Manufacturing  Workforce 
Knowledge  and  Manpower 
Gaps  May  Impede 
Implementation  of  MRLs 


The  services  are  in  the  beginning  stages  of  revitalizing  their  manufacturing 
workforce,  largely  in  response  to  a  February  2006  Defense  Science  Board 
task  force  report  on  “The  Manufacturing  Technology  Program:  A  Key  to 
Affordably  Equipping  the  Future  Force.”  The  report  acknowledged  that 
both  the  manufacturing  expertise  in  the  workforce  and  program  funding 
have  declined,  thus  eliminating  much  of  the  engineering  and 
manufacturing  talent  across  DOD  and  the  industrial  base.  The  report 
concluded  that  what  was  once  a  promising  career  field  in  the  military 
services — with  promotion  paths,  training,  and  professional  development — 
has  been  systematically  eliminated  over  the  past  few  decades.  Table  4 
shows  the  decrease  in  the  manufacturing  career  field  across  DOD  from 
2001  to  2007. 


Table  4:  Percent  of  Manufacturing  Workforce  Decrease  from  2001  to  2007 


Percent 


2001 

2002 

2003 

2004 

2005 

2006 

2007 

reduction 

Army 

2,427 

2,333 

2,215 

2,226 

2,287 

2,193 

2,083 

14 

Navy 

1,997 

2,297 

2,259 

2,232 

2,032 

2,000 

1,960 

<1 

Air  Force 

518 

499 

409 

408 

407 

334 

326 

37 

Source:  DOD. 

Note:  Data  include  military  and  civilian  personnel. 


As  indicated,  DOD’s  manufacturing  career  workforce  trends  show  an 
overall  decline,  with  the  Army  and  Air  Force  having  had  the  biggest 
declines  at  14  percent  and  37  percent,  respectively.  According  to  a  DCMA 
official,  the  agency  experienced  about  a  30  percent  decrease  during  the 
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Conclusions 


same  timeframe.  An  Army  official  responsible  for  workforce  planning 
activities  noted,  however,  there  are  no  positions  designated  specifically  for 
manufacturing,  which  make  it  difficult  to  determine  the  true  career 
workforce  numbers  in  this  category.  Fewer  experts  mean  that  fewer 
people  at  both  the  working  level  and  in  leadership  positions  understand 
the  processes  involved  in  developing  and  manufacturing  defense  systems 
and  their  importance  in  producing  high-quality  and  reliable  systems. 
Further,  fewer  people  are  capable  of  conducting  production-readiness 
reviews,  evaluating  industry’s  work  on  programs,  and  staying  abreast  of 
industry  research  and  development.  According  to  a  recent  study,  of  major 
concern  is  that  recent  estimates  show  30  percent  of  the  civilian 
manufacturing  workforce — classified  as  production,  quality,  and 
manufacturing — are  eligible  for  full  retirement,  and  approximately  26 
percent  will  become  eligible  for  full  retirement  over  the  next  4  years.  This 
means  DOD  will  soon  have  an  exodus  of  its  manufacturing  workforce  and, 
accordingly,  must  plan  for  this  eventuality. 

Although  the  services  are  at  the  beginning  stages  of  revitalizing  their 
production  and  manufacturing  workforce,  program  officials  believe  they 
currently  do  not  have  the  in-house  expertise  with  the  requisite  knowledge 
to  assess  manufacturing,  if  MRLs  were  to  be  mandated  and  deployed 
across  DOD.  For  example,  in  interviews  with  career  planning  officials  at 
the  military  services,  most  report  that  they  have  workforce  challenges  in 
manufacturing  knowledge  gaps  or  insufficient  number  of  personnel  to 
conduct  the  work,  or  both.  The  Defense  Contract  Management  Agency 
reported  similar  manufacturing  knowledge  gaps  due  to  a  lack  of  focus  in 
this  area,  but  it  now  has  new  leadership  in  place  and  is  establishing  plans 
to  address  these  deficiencies.  Essentially,  these  knowledge  deficiencies 
affect  many  areas,  such  as  policy  support  for  programs,  the  ability  to 
develop  an  effective  strategic  plan  and  investment  strategy  for 
manufacturing  technology,  the  ability  to  implement  MRLs  and  conduct 
assessments,  and  the  ability  to  effectively  and  affordably  acquire  high- 
quality  weapon  systems. 


MRLs,  resourced  and  used  effectively,  offer  the  potential  for  DOD  to 
achieve  substantial  savings  and  efficiencies  in  developing  and  acquiring 
weapon  systems.  MRLs  have  been  shown  to  work  in  reducing  the  cost  and 
time  for  developing  technologies  and  producing  systems.  Moreover,  they 
have  been  shown  to  work  on  individual  programs,  and  some  Army 
commands  and  Air  Force  centers  have  adopted  them.  They  are  consistent 
with  commercial  best  practices  and  have  even  been  adopted  by  some 
defense  firms.  Yet,  they  have  not  been  adopted  DOD-wide.  MRLs  are  being 
met  with  resistance  similar  to  that  experienced  by  technology  readiness 
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levels  when  they  were  first  introduced.  However,  technology  readiness 
levels  are  now  widely  accepted  and  used  across  DOD. 

While  MRLs  represent  a  common  body  of  knowledge  and  reflect  many  of 
the  practices  used  by  leading  commercial  companies,  there  is  room  for 
improvement.  Criteria  used  for  getting  manufacturing  processes  under 
control  are  still  not  specific  enough,  allowing  demonstration  of  controls  to 
occur  too  late  in  the  process — after  the  milestone  C  decision  authorizing 
low-rate  initial  production — whereas  commercial  firms  require  that  critical 
processes  be  in  control  earlier.  While  MRLs  represent  positive  change, 
unless  these  criteria  are  strengthened  at  the  time  a  production  decision  is 
made,  DOD  will  have  missed  an  opportunity  to  reduce  the  risk  of 
continued  cost  growth  on  acquisition  programs.  Moreover,  use  of  MRLs 
would  be  enhanced  by  the  development  of  analytical  tools,  such  as  those 
used  by  Honeywell,  to  support  MRL  assessments. 

A  serious  concern  is  that  DOD’s  in-house  manufacturing  workforce  has 
been  diminishing  for  decades  and  that,  therefore,  could  hamper  successful 
implementation  of  MRLs.  Unless  DOD  develops  long-range  plans  to  build 
its  in-house  manufacturing  workforce,  it  may  not  be  able  to  realize  the  full 
potential  of  integrating  manufacturing  readiness  levels  into  its  processes. 


Recommendations  for  ensure  that  DOD  is  taking  steps  to  strengthen  and  improve  the 

producibility  and  manufacturing  readiness  of  technologies,  weapon 
Executive  Action  systems,  subsystems,  or  manufacturing  processes,  we  recommend  that  the 

Secretary  of  Defense  do  the  following: 

•  Require  the  assessment  of  manufacturing  readiness  across  DOD 
programs  using  consistent  MRL  criteria  as  basis  for  measuring, 
assessing,  reporting,  and  communicating  manufacturing  readiness  and 
risk  on  science  and  technology  transition  projects  and  acquisition 
programs. 

•  Direct  the  Office  of  the  Director,  Defense  Research  and  Engineering  to 
examine  strengthening  the  MRL  criteria  related  to  the  process 
capability  and  control  of  critical  components  and/or  interfaces  prior  to 
milestone  C,  or  equivalent,  for  low-rate  initial  production  decision. 

•  Direct  the  Office  of  the  Director,  Defense  Research  and  Engineering  to 
assess  the  need  for  analytical  models  and  tools  to  support  MRL 
assessments. 
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•  Assess  the  adequacy  of  the  manufacturing  workforce  knowledge  and 
skills  base  across  the  military  services  and  defense  agencies  and 
develop  a  plan  to  address  current  and  future  workforce  gaps. 


Agency  Comments 
and  Our  Evaluation 


DOD  provided  us  written  comments  on  a  draft  of  this  report.  DOD 
partially  concurred  with  our  recommendation  to  require  the  assessment  of 
manufacturing  readiness  across  DOD  programs  using  MRL  criteria,  and 
concurred  with  our  other  recommendations.  Their  comments  can  be 
found  in  appendix  IV  of  this  report. 

In  its  comments,  DOD  partially  concurred  with  the  recommendation  that 
DOD  programs  be  required  to  assess  manufacturing  readiness  using 
consistent  MRL  criteria  as  the  basis  for  measuring,  assessing,  reporting, 
and  communicating  manufacturing  readiness  and  risk  on  science  and 
technology  transition  projects  and  acquisition  programs.  DOD  cites  the 
Department  of  Defense  Instruction  5000.02  as  reflecting  on  manufacturing 
throughout  the  acquisition  life  cycle  and,  specifically,  establishing  a 
framework  to  continually  assess  and  mitigate  manufacturing  risks.  In  its 
remarks,  DOD  states  that  the  manufacturing  readiness  criteria  will  be 
tailored  to  programs  and  embedded  into  reviews  and  assessment 
templates,  including  systems  engineering  reviews,  preliminary  design 
reviews,  and  critical  design  reviews  as  well  as  acquisition  phase  exit 
criteria. 


While  we  are  encouraged  by  DOD’s  plans  to  incorporate  manufacturing 
readiness  criteria  into  various  assessments,  we  are  concerned  about  the 
absence  of  any  reference  to  MRLs,  which  identify  specific  benchmarks  for 
each  acquisition  phase.  It  is  unclear  from  DOD’s  comments  whether  it 
intends  to  use  a  common  definition  of  manufacturing  readiness  as 
acquisition  phase  exit  criteria  or  whether  the  exit  criteria  will  be  decided 
on  a  case-by-case  basis.  While  tailoring  to  individual  programs  is 
appropriate,  tailoring  must  take  place  in  the  context  of  well-understood 
criteria  for  moving  from  phase  to  phase.  A  hallmark  of  the  commercial 
programs  we  have  looked  at  in  this  and  other  reviews  is  the  reliance  on 
disciplined  processes  for  assessing  readiness  to  proceed  into  more  costly 
development  and  production  phases.  Firm  criteria  are  needed  to  identify 
and  address  producibility  and  manufacturing  risks  on  a  timely  basis, 
before  they  result  in  expensive  production  problems. 

We  also  received  technical  comments  from  DOD,  which  have  been 
addressed  in  the  report,  as  appropriate. 
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We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense;  the 
Secretary  of  the  Army;  Secretary  of  the  Navy;  Secretary  of  the  Air  Force; 
Director,  Missile  Defense  Agency;  Director,  Defense  Contract  Management 
Agency;  and  Office  of  Management  and  Budget.  In  addition,  the  report  will 
be  made  available  at  no  charge  on  the  GAO  Web  site  at 
http://www.gao.gov. 

If  you,  or  your  staff,  have  any  questions  concerning  this  report,  please 
contact  me  at  (202)  512-4841.  Contact  points  for  our  offices  of 
Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  The  major  contributors  are  listed  in  appendix  V. 


Michael  J.  Sullivan 
Director 

Acquisition  and  Sourcing  Management 
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Appendix  I:  Scope  and  Methodology 


This  report  compares  the  Department  of  Defense  (DOD)  and  its  large 
prime  contractors’  manufacturing  practices  with  those  of  leading 
commercial  companies — with  a  focus  on  improving  the  manufacturing  of 
defense  weapon  systems.  Specifically,  we  assessed  (1)  the  manufacturing 
problems  experienced  by  DOD,  (2)  how  manufacturing  readiness  levels 
(MRLs)  can  address  DOD’s  manufacturing  problems,  (3)  how  proposed 
MRLs  compare  to  manufacturing  best  practices  of  leading  commercial 
companies,  and  (4)  the  challenges  and  barriers  to  implementing  MRLs  at 
DOD. 

To  identify  the  manufacturing  problems  experienced  by  DOD,  we 
performed  an  aggregate  analysis  of  DOD  programs  from  our  annual 
assessment  database.  We  also  conducted  case  studies  of  four  programs 
with  known  cost  and  schedule  problems  to  make  observations  on  the 
types  of  problems  DOD  weapon  systems  may  experience.  The  programs 
we  reviewed,  along  with  the  prime  contractors  responsible  for  developing 
the  systems,  are 

•  Joint  Air-to-Surface  Standoff  Missile,  an  air-to-surface  missile  funded 
by  the  Air  Force  and  developed  by  Lockheed  Martin; 


•  Exoatmospheric  Kill  Vehicle,  a  ballistic-missile  interceptor  funded 
by  the  Missile  Defense  Agency  and  developed  by  Raytheon; 


•  Electromagnetic  Aircraft  Launch  System,  a  launch  system  for 
aircraft  carriers  funded  by  the  Navy  and  developed  by  General 
Atomics;  and 


•  H-l  helicopter  upgrade,  tactical  utility  and  attack  helicopters  funded 
by  the  Navy  and  developed  by  Bell  Helicopter. 

To  evaluate  the  four  DOD  weapon  programs,  we  examined  program 
documentation,  such  as  acquisition  decision  memos  and  production 
readiness  reviews,  and  held  discussions  with  manufacturing  and  systems 
engineering  officials  from  DOD  program  offices,  the  prime  contractors, 
and  the  Defense  Contract  Management  Agency.  Based  on  the  information 
gathered  through  interviews  conducted  and  documentation  synthesized, 
we  identified  commonalities  among  the  case  studies. 

To  determine  how  MRLs  can  address  the  manufacturing  problems 
experienced  by  defense  programs,  we  conducted  interviews  with  officials 
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from  the  Office  of  Secretary  of  Defense,  Office  of  the  Director,  Defense 
Research  &  Engineering,  Joint  Defense  Manufacturing  Technology  Panel 
working  group,  National  Center  for  Advanced  Technologies,  National 
Defense  Industrial  Association,  and  Defense  Acquisition  University  on 
their  observations  on  MRLs.  We  also  reviewed  the  MRL  deskbook,  matrix 
(risk  areas),  analyses,  and  training  materials.  We  also  conducted 
interviews  with  Army,  Navy,  and  Air  Force  officials  who  were  involved  or 
familiar  with  the  pilot  tests  of  MRLs  on  various  programs.  The  pilot 
programs  we  examined  at  the  military  services  include  the  following 

•  Army — micro  electro-mechanical  systems  inertial  measurement  unit, 
micro  electro-mechanical  systems  safety  arm,  ferroelectric  and  micro 
electro-mechanical  systems  phase  shifter,  low-cost  materials  for 
improved  protection,  rotorcraft  cabin  floor  structure,  embedded 
sensors,  and  armor  manufacturing; 


•  Air  Force — MQ-9  Reaper,  F-35  Joint  Strike  Fighter  induct  inlet,  high- 
durability  hot  exhaust  structures,  F-135  Pratt  &  Whitney  propulsion 
system,  sensor  hardening  for  tactical  systems,  and  X-Band  thin  array 
radar;  and 


•  Navy — P-8A  aircraft. 

To  identify  practices  and  criteria  used  by  leading  commercial  companies 
that  can  be  used  to  improve  DOD’s  manufacturing  process,  we  selected 
and  visited  five  companies  based  on  several  criteria:  companies  that  (1) 
make  products  that  are  comparable  to  DOD  in  terms  of  complexity,  (2) 
are  recognized  as  leaders  in  developing  manufacturing  readiness  criteria, 
or  (3)  have  won  awards  for  their  manufacturing  best  practices,  or  a  mix  of 
the  above.  We  met  with  these  companies  to  discuss  their  product- 
development  and  manufacturing  practices  and  the  steps  that  they  take  to 
mitigate  manufacturing  risks,  ensure  manufacturing  readiness,  and 
improve  supplier  quality.  We  met  with  these  companies  to  discuss  their 
product-development  life  cycle  and  the  methods  and  metrics  they  use  to 
measure  manufacturing  maturity  and  producibility;  manufacturing  risk 
management;  supplier  management;  and  the  key  factors  in  the  company’s 
successful  manufacturing  outcomes.  We  generalized  much  of  the 
information  due  to  the  proprietary  nature  of  the  data  relating  to  their 
manufacturing  processes.  Several  companies  provided  data  on  specific 
processes  or  products  that  they  agreed  to  allow  us  to  include  in  this 
report.  We  reported  on  four  of  the  five  companies  we  visited.  The  five 
companies  we  visited  include  the  following 
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•  GE  Aviation,  a  leading  aerospace  company,  whose  portfolio  includes 
commercial  engines  and  services,  military  engines  and  services, 
business  and  general  aviation,  engine  components,  and  aviation 
systems.  We  met  with  manufacturing  and  quality  officials  in  Cincinnati, 
Ohio,  and  discussed  their  manufacturing  practices  and  manufacturing 
maturity  metrics.  We  also  toured  their  Lean  Lab  production  facility  and 
saw  how  these  practices  were  applied. 


•  GE  Healthcare,  which  manufactures  a  range  of  products  and  services 
that  includes  medical  imaging  and  information  technologies  and 
medical  diagnostics.  We  met  with  manufacturing  officials  at  their 
Milwaukee,  Wisconsin,  plant  and  discussed  their  manufacturing 
practices,  including  the  development  and  manufacturing  of  their 
Gemstone  scintillator  for  use  on  advanced  CT  scanners. 


•  Honeywell  Aerospace,  a  global  provider  of  integrated  avionics, 
engines,  systems,  and  services  for  aircraft  manufacturers,  airlines, 
business  and  general  aviation,  and  military  and  space  operations.  We 
met  with  manufacturing  officials  at  their  Phoenix,  Arizona,  facility  and 
discussed  their  manufacturing  maturity  processes  and  the  models  and 
tools  they  used  to  assess  this. 


•  Siemens  Mobility,  a  division  of  Siemens  that  develops  and  builds 
light  rail  cars  for  the  North  American  market.  We  met  with 
manufacturing  and  procurement  officials  at  their  Sacramento, 
California,  manufacturing  and  assembly  plant  to  discuss  the 
manufacturing  processes  used  in  building  their  rail  cars  and  their 
supplier  management  practices. 


•  Toyota  Motor  Engineering  and  Manufacturing  is  responsible  for 
Toyota’s  engineering  design,  development  and  manufacturing  activities 
in  North  America.  We  met  with  officials  in  their  production  engineering 
division  in  Erlanger,  Kentucky,  and  also  in  their  Toyota  Technical 
Center  located  in  Ann  Arbor,  Michigan,  and  discussed  their  vehicle 
development  process  and  their  methods  for  assuring  supplier  quality. 

At  each  of  the  companies,  we  interviewed  senior  management  officials 
knowledgeable  about  the  manufacturing  methods,  techniques,  and 
practices  used  throughout  manufacturing  and  product  development  to 
ensure  manufacturing  maturity  and  producibility  of  their  products.  In 
particular,  we  discussed  their  (1)  product  development  life  cycle  and  the 
methods,  metrics,  and  tools  used  to  determine  manufacturing  maturity 
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and  producibility,  (2)  methods  for  identifying  and  mitigating  risks  in 
manufacturing  a  product,  and  (3)  methods  for  supplier  management  to 
provide  steady  supply  of  quality  parts. 

In  addition,  we  compared  the  practices  of  commercial  firms  to  two  major 
defense  weapon  systems  known  to  be  producing  systems  within  cost  and 
schedule  goals  and  with  successful  manufacturing  outcomes.  To  evaluate 
these  two  programs,  we  examined  program  documentation  and  held 
discussions  with  program  and  contracting  officials.  The  two  systems  we 
reviewed,  along  with  the  prime  contractors  responsible  for  developing  the 
systems,  are 

•  Lakota  aircraft,  a  light  utility  helicopter  that  conducts  noncombat 
missions,  funded  by  the  Army  and  developed  by  the  European 
Aeronautic  Defence  and  Space  Company;  and 


•  Standard  Missile  3  Block  1A,  a  ship-based  antiballistic  missile, 
funded  by  the  Missile  Defense  Agency  and  developed  by  Raytheon. 

To  determine  the  challenges  and  barriers  to  MRL  implementation  efforts, 
we  interviewed  officials  who  were  involved  with  the  draft  policy  to 
standardize  MRLs,  as  well  as  the  military-service  policy  organizations  that 
commented  on  the  proposal.  We  also  synthesized  the  information 
gathered  at  the  various  levels  throughout  the  defense  community  to 
determine  the  issues  surrounding  MRLs  as  well  as  their  merits.  These 
DOD  organizations  include  the  Office  of  the  Secretary  of  Defense,  each  of 
the  military-service  policy  groups  and  program  offices,  the  Office  of  the 
Director,  Defense  Research  and  Engineering,  Systems  and  Software 
Engineering,  and  Joint  Defense  Manufacturing  Technology  Panel.  To 
obtain  an  understanding  of  the  workforce  challenges  in  manufacturing,  we 
reviewed  selected  documentation — such  as  Defense  Science  Board 
studies — and  interviewed  officials  at  each  of  the  military  services  and 
Defense  Contract  Management  Agency  who  were  responsible  for 
workforce  planning  activities  and  revitalization  initiatives. 

We  conducted  this  performance  audit  from  January  2009  to  February  2010 
in  accordance  with  generally  accepted  government  auditing  standards. 
These  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives. 
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MRL  1 — Basic  Manufacturing  Implications  Identified 

This  is  the  lowest  level  of  manufacturing  readiness.  The  focus  is  to  address 
manufacturing  shortfalls  and  opportunities  needed  to  achieve  program 
objectives.  Basic  research  (i.e.,  budget  activity  6.1  funds)  begins  in  the 
form  of  studies. 

MRL  2 — Manufacturing  Concepts  Identified 

This  level  is  characterized  by  describing  the  application  of  new 
manufacturing  concepts.  Applied  research  (i.e.,  budget  activity  6.2  funds) 
translates  basic  research  into  solutions  for  broadly  defined  military  needs. 
Typically  this  level  of  readiness  in  the  science  and  technology 
environment  includes  identification,  paper  studies,  and  analysis  of 
material  and  process  approaches.  An  understanding  of  manufacturing 
feasibility  and  risk  is  emerging. 

MRL  3 — Manufacturing  Proof  of  Concept  Developed 

This  level  begins  the  validation  of  the  manufacturing  concepts  through 
analytical  or  laboratory  experiments.  This  level  of  readiness  is  typical  of 
technologies  in  the  science  and  technology  funding  categories  of  Applied 
Research  and  Advanced  Development  (i.e.,  budget  activity  6.3  funds). 
Materials  or  processes,  or  both,  have  been  characterized  for 
manufacturability  and  availability  but  further  evaluation  and 
demonstration  is  required.  Experimental  hardware  models  have  been 
developed  in  a  laboratory  environment  that  may  possess  limited 
functionality. 

MRL  4 — Capability  to  Produce  the  Technology  in  a  Laboratory 
Environment 

This  level  of  readiness  is  typical  for  science  and  technology  programs  in 
the  budget  activity  6.2  and  6.3  categories  and  acts  as  exit  criteria  for  the 
materiel  solution  analysis  phase  approaching  a  milestone  A  decision. 
Technologies  should  have  matured  to  at  least  technology  readiness  level  4. 
This  level  indicates  that  the  technologies  are  ready  for  the  technology- 
development  phase  of  acquisition.  At  this  point,  required  investments, 
such  as  manufacturing  technology  development,  have  been  identified. 
Processes  to  ensure  manufacturability,  producibility,  and  quality  are  in 
place  and  are  sufficient  to  produce  technology  demonstrators. 
Manufacturing  risks  have  been  identified  for  prototype  build,  and 
mitigation  plans  are  in  place.  Target  cost  objectives  have  been  established 
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and  manufacturing  cost  drivers  have  been  identified.  Producibility 
assessments  of  design  concepts  have  been  completed.  Key  design 
performance  parameters  have  been  identified  as  well  as  any  special 
tooling,  facilities,  material  handling,  and  skills  required. 

MRL  5 — Capability  to  Produce  Prototype  Components  in  a 
Production-Relevant  Environment 

This  level  of  maturity  is  typical  of  the  midpoint  in  the  technology- 
development  phase  of  acquisition,  or  in  the  case  of  key  technologies,  near 
the  midpoint  of  an  advanced  technology-demonstration  project. 
Technologies  should  have  matured  to  at  least  technology  readiness  level  5. 
The  industrial  base  has  been  assessed  to  identify  potential  manufacturing 
sources.  A  manufacturing  strategy  has  been  refined  and  integrated  with 
the  risk-management  plan.  Identification  of  enabling/critical  technologies 
and  components  is  complete.  Prototype  materials,  tooling  and  test 
equipment,  as  well  as  personnel  skills,  have  been  demonstrated  on 
components  in  a  production-relevant  environment,  but  many 
manufacturing  processes  and  procedures  are  still  in  development. 
Manufacturing  technology  development  efforts  have  been  initiated  or  are 
ongoing.  Producibility  assessments  of  key  technologies  and  components 
are  ongoing.  A  cost  model  has  been  constructed  to  assess  projected 
manufacturing  cost. 

MRL  6 — Capability  to  Produce  a  Prototype  System  or  Subsystem  in 
a  Production-Relevant  Environment 

This  MRL  is  associated  with  readiness  for  a  milestone  B  decision  to  initiate 
an  acquisition  program  by  entering  into  the  engineering  and  manufacturing 
development  phase  of  acquisition.  Technologies  should  have  matured  to  at 
least  technology  readiness  level  6.  It  is  normally  seen  as  the  level  of 
manufacturing  readiness  that  denotes  completion  of  science  and 
technology  development  and  acceptance  into  a  preliminary  system  design. 
An  initial  manufacturing  approach  has  been  developed.  The  majority  of 
manufacturing  processes  have  been  defined  and  characterized,  but  there 
are  still  significant  engineering  or  design  changes,  or  both,  in  the  system 
itself.  However,  preliminary  design  of  critical  components  has  been 
completed  and  producibility  assessments  of  key  technologies  are 
complete.  Prototype  materials,  tooling  and  test  equipment,  as  well  as 
personnel  skills  have  been  demonstrated  on  systems  or  subsystems,  or 
both,  in  a  production-relevant  environment.  A  cost  analysis  has  been 
performed  to  assess  projected  manufacturing  cost  versus  target  cost 
objectives  and  the  program  has  in  place  appropriate  risk  reduction  to 
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achieve  cost  requirements  or  establish  a  new  baseline.  This  analysis 
should  include  design  trades.  Producibility  considerations  have  shaped 
system-development  plans.  Industrial  capabilities  assessment  for 
milestone  B  has  been  completed.  Long-lead  and  key  supply-chain  elements 
have  been  identified.  All  subcontractors  have  been  identified. 

MRL  7 — Capability  to  Produce  Systems,  Subsystems,  or 
Components  in  a  Production-Representative  Environment 

This  level  of  manufacturing  readiness  is  typical  for  the  midpoint  of  the 
engineering  and  manufacturing-development  phase  leading  to  the  post- 
critical  design  review  assessment.  Technologies  should  be  maturing  to  at 
least  technology  readiness  level  7.  System  detailed  design  activity  is 
underway.  Material  specifications  have  been  approved  and  materials  are 
available  to  meet  the  planned  pilot-line  build  schedule.  Manufacturing 
processes  and  procedures  have  been  demonstrated  in  a  production- 
representative  environment.  Detailed  producibility  trade  studies  and  risk 
assessments  are  underway.  The  cost  model  has  been  updated  with 
detailed  designs,  rolled  up  to  system  level,  and  tracked  against  allocated 
targets.  Unit-cost  reduction  efforts  have  been  prioritized  and  are 
underway.  The  supply  chain  and  supplier  quality  assurance  have  been 
assessed  and  long-lead  procurement  plans  are  in  place.  Production  tooling 
and  test  equipment  design  and  development  have  been  initiated. 

MRL  8 — Pilot-Line  Capability  Demonstrated;  Ready  to  Begin  Low- 
Rate  Initial  Production 

This  level  is  associated  with  readiness  for  a  milestone  C  decision,  and 
entry  into  low-rate  initial  production.  Technologies  should  have  matured 
to  at  least  technology  readiness  level  7.  Detailed  system  design  is 
essentially  complete  and  sufficiently  stable  to  enter  low-rate  production. 
All  materials  are  available  to  meet  the  planned  low-rate  production 
schedule.  Manufacturing  and  quality  processes  and  procedures  have  been 
proven  in  a  pilot-line  environment  and  are  under  control  and  ready  for 
low-rate  production.  Known  producibility  risks  pose  no  significant 
challenges  for  low-rate  production.  The  engineering  cost  model  is  driven 
by  detailed  design  and  has  been  validated  with  actual  data.  The  Industrial 
Capability  Assessment  for  milestone  C  has  been  completed  and  shows  that 
the  supply  chain  is  established  and  stable. 
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MRL  9 — Low-Rate  Production  Demonstrated;  Capability  in  Place  to 
Begin  Full-Rate  Production 

At  this  level,  the  system,  component,  or  item  has  been  previously 
produced,  is  in  production,  or  has  successfully  achieved  low-rate  initial 
production.  Technologies  should  have  matured  to  at  least  technology 
readiness  level  9.  This  level  of  readiness  is  normally  associated  with 
readiness  for  entry  into  full-rate  production.  All  systems- 
engineering/design  requirements  should  have  been  met  such  that  there  are 
minimal  system  changes.  Major  system  design  features  are  stable  and  have 
been  proven  in  test  and  evaluation.  Materials  are  available  to  meet  planned 
rate  production  schedules.  Manufacturing  process  capability  in  a  low-rate 
production  environment  is  at  an  appropriate  quality  level  to  meet  design 
key-characteristic  tolerances.  Production  risk  monitoring  is  ongoing.  Low- 
rate  initial  production  cost  targets  have  been  met,  with  learning  curves 
validated.  The  cost  model  has  been  developed  for  the  full-rate  production 
environment  and  reflects  the  effect  of  continuous  improvement. 

MRL  10 — Full-Rate  Production  Demonstrated  and  Lean  Production 
Practices  in  Place 

This  is  the  highest  level  of  production  readiness.  Technologies  should  have 
matured  to  at  least  technology  readiness  level  9.  This  level  of 
manufacturing  is  normally  associated  with  the  production  or  sustainment 
phases  of  the  acquisition  life  cycle.  Engineering/design  changes  are  few 
and  generally  limited  to  quality  and  cost  improvements.  System, 
components,  or  items  are  in  full-rate  production  and  meet  all  engineering, 
performance,  quality,  and  reliability  requirements.  Manufacturing  process 
capability  is  at  the  appropriate  quality  level.  All  materials,  tooling, 
inspection  and  test  equipment,  facilities,  and  manpower  are  in  place  and 
have  met  full-rate  production  requirements.  Rate  production  unit  costs 
meet  goals,  and  funding  is  sufficient  for  production  at  required  rates.  Lean 
practices  are  well  established  and  continuous  process  improvements  are 
ongoing. 


Page  54 


GAO-10-439  DOD  Manufacturing  Readiness 


Appendix  III:  Manufacturing  Readiness  Level 
(MRL)  Threads  and  Subthreads  (Risk  Areas) 


Acquisition 

Phase 


Pre-material 
solution  analysis 


Material 

solution 

analsysis 


Technology 

development 


Engineering  and 
manufacturing 
development 


Low  rate 
initial 
product 


Full 

rate 

product 


Thread 

Sub¬ 

thread 

MRL  1 

MRL  2 

MRL  3 

MRL  4  ^ 

^  MRL  5 

MRL  6  ^ 

^MRL7 

MRL  8^ 

^MRL9 

^  MRL  10 

- 

Technology 

maturity 

Should  be 
assessed 
atTRL  1 

Should  be 
assessed 
atTRL  2 

Should  be 
assessed 
atTRL  3 

Should  be 
assessed 
atTRL  4 

Should  be 
assessed 
atTRL  5 

Should  be 
assessed 
atTRL  6 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  8 

Should  be 
assessed 
atTRL  9 

Technol¬ 
ogy  and 
industrial 
base 

Technology 
transition 
to  produc¬ 
tion 

Potential 

sources 

identified 
for  technol¬ 
ogy  needs. 
Understand 
state  of  the 
art. 

Industrial  base 
capabilities 
surveyed  and 
known;  gaps/ 
risks  identified 
for  preferred 
concept,  key 
technologies, 
components, 
and/or  key 
processes. 

Industrial  base 
assessment 
initiated  to 
identify  poten¬ 
tial  manufac¬ 
turing  sources. 
Sole/  single/ 
foreign  source 
vendors  have 
been  identified 
and  planning 
has  begun  to 
minimize  risks. 

Industrial  Capa¬ 
bilities  Assess¬ 
ment  (ICA)  for 
Milestone  (MS) 

B  has  been  com¬ 
pleted.  Indus¬ 
trial  capability  in 
place  to  support 
manufacturing 
of  development 
articles.  Plans  to 
minimize  sole/ 
foreign  sources 
complete.  Need 
for  sole/single/ 
foreign  sources 
justified.  Poten¬ 
tial  alternative 
sources  identi¬ 
fied. 

Industrial 
capabil¬ 
ity  to  support 
production 
has  been  ana¬ 
lyzed.  Sole/ 
single/ for¬ 
eign  sources 
stability  is 
assessed/ 
monitored. 
Developing 
potential 
alternate 

sources  as 

necessary. 

ICA  for  MS 

C  has  been 
completed. 
Industrial 
capability 
is  in  place 
to  support 
Low  Rate 
Initial 
Produc¬ 
tion  (LRIP). 
Sources  are 
available, 
multi¬ 
sourcing 
where  cost- 
effective  or 
necessary 
to  mitigate 
risk. 

Industrial 
capability 
is  in  place 
to  support 
start  of  Full 
Rate  Produc¬ 
tion  (FRP). 

Industrial 

capability 

supports 

FRP  and  is 
assessed 
to  support 
modifica¬ 
tions, 
upgrades, 
surge 
and  other 
potential 
manufactur¬ 
ing  require¬ 
ments. 

Technol¬ 
ogy  and 
industrial 
base 

Manufac¬ 

turing 

technology 

develop¬ 

ment 

New  manu¬ 
facturing 
concepts 
and 

potential 

solutions 

identified. 

Manu¬ 
facturing 
technology 
concepts 
identified 
through  ex¬ 
periments/ 
models. 

Manufactur¬ 
ing  Science 
and  Advanced 
Manufacturing 
Technology 
requirements 
identified. 

Required 
manufacturing 
technology 
development 
efforts  initiated 
as  applicable. 

Manufacturing 
technology 
efforts  continu¬ 
ing.  Required 
manufacturing 
technology 
development 
solutions  dem¬ 
onstrated  in  a 
production-rele¬ 
vant  environ¬ 
ment. 

Manufactur¬ 
ing  technol¬ 
ogy  efforts 
continuing. 
Required 
manu¬ 
facturing 
technology 
development 
solutions 
demon¬ 
strated  in  a 
production- 
representative 
environment. 

Primary 
manu¬ 
facturing 
technol¬ 
ogy  efforts 
concluding 
and  some 
improve¬ 
ment 
efforts 
continuing. 
Required 
manu¬ 
facturing 
technology 
solutions 
validated 
on  a  pilot 
line. 

Manufactur¬ 
ing  technol¬ 
ogy  process 
improve¬ 
ment  efforts 
initiated  for 
FRP. 

Manu¬ 
facturing 
technology 
continuous 
process  im¬ 
provements 
ongoing. 

Design 

Producibil- 
ity  program 

Relevant 
materials/ 
processes 
evalu¬ 
ated  for 
manufactur¬ 
ability  using 

Initial  produc- 
ibility  and 
manufac¬ 
turability 
assessment  of 
preferred  sys¬ 
tems  concepts 

Producibility 
and  manufac¬ 
turability 
assessments 
of  key  tech¬ 
nologies  and 
components 

Producibility  as¬ 
sessments  and 
producibility 
trade  studies 
(performance 
vs.  produc¬ 
ibility)  of  key 

Detailed 
produc¬ 
ibility  trade 
studies  using 
knowledge 
of  key  design 
characteristics 

Produc¬ 
ibility 
improve¬ 
ments 
imple¬ 
mented 
on  system. 

Prior 
produc¬ 
ibility  im¬ 
provements 
analyzed 
for  ef¬ 
fectiveness 

Design 
producibil¬ 
ity  improve¬ 
ments  dem¬ 
onstrated  in 
FRP.  Process 

Page  55 


GAO-10-439  DOD  Manufacturing  Readiness 


Appendix  III:  Manufacturing  Readiness  Level 
(MRL)  Threads  and  Subthreads  (Risk  Areas) 


Acquisition 

Phase 


Pre-material 
solution  analysis 


Material 

solution 

analsysis 


Technology 

development 


Engineering  and 
manufacturing 
development 


Low  rate 
initial 
product 


Full 

rate 

product 


Sub- 

A 

,  A 

Thread 

thread 

MRL  1 

MRL  2 

MRL  3 

MRL  4  i 

^  MRL  5 

MRL  6  V 

k  MRL  7 

MRL8> 

k  MRL  9 

MRL  10 

Technology 

maturity 

Should  be 

Should  be 

Should  be 

Should  be 

Should  be 

Should  be 

Should  be 

Should  be 

Should  be 

Should  be 

— 

assessed 

assessed 

assessed 

assessed 

assessed 

assessed 

assessed 

assessed 

assessed 

assessed 

atTRL  1 

atTRL  2 

atTRL  3 

atTRL  4 

atTRL  5 

atTRL  6 

atTRL  7 

atTRL  7 

atTRL  8 

atTRL  9 

experimen- 

completed. 

initiated  as  ap- 

technologies/ 

and  related 

Known 

during  LRIP. 

produc- 

tal  results. 

Results  con- 

propriate.  On- 

components 

manufactur- 

producibil- 

Producibility 

ibility  im- 

sidered  in 

qoinq  desiqn 

completed. 

ing  process 

ity  issues 

issues/  risks 

provements 

selection  of 

trades  consider 

Results  used  to 

capability 

have  been 

discovered 

onqoinq.  All 

preferred  de- 

manufactur- 

shape  Acquisi- 

completed. 

resolved 

in  LRIP 

modifica- 

sign  concepts 

ing  processes 

tion  Strategy 

Producibility 

and  pose 

have  been 

tions, 

and  reflected 

and  industrial 

(AS),  Systems 

enhancement 

no  signifi- 

mitigated 

upgrades, 

in  Technology 

base  capabil- 

Engineering 

efforts  (e.g., 

cant  risk  for 

and  pose  no 

Diminishing 

Development 

ity  constraints. 

Plan  (SEP), 

DFX)  onqoinq 

LRIP. 

significant 

Manufactur- 

Strategy  (TDS) 

Manufactur- 

Manufacturing 

for  optimized 

risk  for  FRP. 

ing  Sources 

key  com- 

ing  processes 

and  Produc- 

integrated 

and  Material 

ponents/ 

assessed  for 

ibility  plans, and 

system. 

Shortages 

technologies. 

capability  to 

planning  for 

Manufactur- 

(DMSMS), 

test  and  verify 

Engineering  and 

ing  processes 

and  other 

in  production, 

Manufacturing 

reassessed  as 

changes 

and  influence 

Development 

needed  for 

assessed  for 

on  operations 

(EMD)  or  tech- 

capability  to 

produc- 

&  support. 

nology  insertion 

test  and  verify 

ibility. 

programs. 

potential  in- 

Preliminary 

fluence  on 

design  choices 

operations  & 

assessed  against 
manufactur- 

support. 

ing  processes 
and  industrial 

base  capabil¬ 
ity  constraints. 
Producibility 
enhancement 
efforts  (e.g., 
Design  for 
Manufacturing 
(DFX))  initiated. 

Design 

Design 

Manufactur- 

Applica- 

Top 

SEP  andTest 

Lower  level 

System  al- 

Product 

Detailed 

Major  prod- 

Product 

maturity 

ing  research 

tions 

level  perfor- 

and  Evaluation 

performance 

located  baseline 

requirements 

design  of 

uct  design 

design 

opportuni- 

defined. 

mance  re- 

(T&E)  Strategy 

requirements 

established. 

and  features 

product 

features  and 

is  stable. 

ties  identi- 

Broad  per- 

quirements 

recognize  the 

sufficient  to 

System  and 

are  defined 

features 

configura- 

Design 

fied. 

formance 

defined. 

need  for  the 

proceed  to 

subsystem 

well  enough 

and  inter- 

tion  are 

changes 

goals 

Tradeoffs 

establishment/ 

preliminary 

preliminary 

to  support 

faces  is 

stable.  Sys- 

are  few  and 

identified 

in  design 

validation  of 

design.  All 

design  sufficient 

Critical  De- 

complete. 

tern  design 

generally 

that  may 

options 

manufacturing 

enabling/ 

for  EMD.  All 

sign  Review 

All  product 

has  been 

limited  to 

drive  manu- 

assessed 

capability  and 

critical  tech- 

enabling/critical 

(CDR)  even 

data 

validated 

those  re- 

facturing 

based  on 

management 

nologies  and 

technologies/ 

though  de- 

essential 

through 

quired  for 

options. 

experi- 

of  manufactur- 

components 

components 

sign  chanqe 

for  system 

operational 

continuous 

ments. 

ing  risk  for  the 

identified  and 

have  been 

traffic  may  be 

manu- 

testing  of 

improve- 

Product  life 

product  life 

product  life  cy- 

demonstrated. 

significant.  All 

facturing 

LRIP  items. 

ment  or  in 

cycle  and 

cycle.  Initial 

cle  considered. 

Preliminary  de- 

product  data 

has  been 

Physical 

reaction  to 

potential  Key 

Evaluation  of 

sign  KCs  defined. 

essential  for 

released. 
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Appendix  III:  Manufacturing  Readiness  Level 
(MRL)  Threads  and  Subthreads  (Risk  Areas) 


Acquisition 

Phase 


Pre-material 
solution  analysis 


Material 

solution 

analsysis 


Technology 

development 


Engineering  and 
manufacturing 
development 


Low  rate 
initial 
product 


Full 

rate 

product 


Thread 

Sub¬ 

thread 

MRL  1 

MRL  2 

MRL  3 

MRL  4  ^ 

^  MRL  5 

MRL  6  ^ 

^MRL7 

MRL  8^ 

^MRL9 

^  MRL  10 

- 

Technology 

maturity 

Should  be 
assessed 
atTRL  1 

Should  be 
assessed 
atTRL  2 

Should  be 
assessed 
atTRL  3 

Should  be 
assessed 
atTRL  4 

Should  be 
assessed 
atTRL  5 

Should  be 
assessed 
atTRL  6 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  8 

Should  be 
assessed 
atTRL  9 

technical  re¬ 
quirements 
evaluated. 

Performance 
Parameter 
(KPPs) 
identified 
for  preferred 
systems  con¬ 
cept.  System 
characteristics 
and  measures 
to  support 
required  capa¬ 
bilities  identi¬ 
fied.  Form,  fit, 
and  function 
constraints 
identified,  and 
manufacturing 
capabilities 
identified  for 
preferred  sys¬ 
tems  concepts. 

design  Critical 
Characteristic 
(KCs)  initiated. 
Product  data 
required  for 
prototype 
component 
manufacturing 
released. 

component 
manufactur¬ 
ing  has  been 
released. 
Potential  KC 
risk  issues 
have  been 
identified  and 
mitigation 
plan  is  in 
place. 

Design 
change 
traffic  does 
not  sig¬ 
nificantly 
impact 

LRIP.  KCs 
are  attain¬ 
able  based 
upon 
pilot  line 
demonstra¬ 
tions. 

Configura¬ 
tion  Audit 
(PCA)  or 
equivalent 
complete  as 
necessary. 
Design 
change  traf¬ 
fic  is  limited. 
All  KCs  are 
controlled 
in  LRIP  to 
appropri¬ 
ate  quality 
levels. 

obsoles¬ 
cence.  All 

KCs  are 
controlled 
in  FRP  to 
appropri¬ 
ate  quality 
levels. 

Cost  and- 
funding 

Production 
cost  knowl¬ 
edge  (cost 
modeling) 

Cost  model 

approach 

defined. 

Initial  cost 
targets 
and  risks 
identified. 
High  level 
process 
chart  model 
developed. 
Technology 
cost  models 
developed 
for  new 
process 
steps  and 
materials 
based  on 
experi¬ 
ments. 

Manufacturing, 
material  and 
special  require¬ 
ment  cost 
drivers  identi¬ 
fied.  Detailed 
process  chart 
cost  models 
driven  by  pro¬ 
cess  variables. 
Cost  driver 
uncertainty 
quantified. 

Prototype 
components 
produced  in 
a  produc¬ 
tion  relevant 
environment, 
or  simulations 
drive  end- 
to-end  cost 
models.  Cost 
model  includes 
materials, 
labor,  equip¬ 
ment,  tooling/ 
(Special  Test 
Equipment 
(STE),  setup, 
yield/  scrap/ 
rework,  Work  in 
Process  (WIP), 
and  capabil¬ 
ity/capacity 
constraints). 

Cost  model 
updated  with 
design  require¬ 
ments,  material 
specifications, 
tolerances,  inte¬ 
grated  master 
schedule,  results 
of  system/ 
subsystem 
simulations  and 
production  rel¬ 
evant  prototype 
demonstrations. 

Cost  model 
updated  with 
the  results 
of  systems/ 
sub-systems 
produced  in 
a  production- 
representative 
environment 
and  with  pro¬ 
duction  plant 
layout  and 
design  and 
obsolescence 
solutions. 

Cost 
models 
updated 
with  results 
of  pilot  line 
build. 

FRP  cost 
model  up¬ 
dated  with 
result  of 

LRIP  build. 

Cost  model 
validated 
against 
actual  FRP 
cost. 
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Appendix  III:  Manufacturing  Readiness  Level 
(MRL)  Threads  and  Subthreads  (Risk  Areas) 


Acquisition 

Phase 


Pre-material 
solution  analysis 


Material 

solution 

analsysis 


Technology 

development 


Engineering  and 
manufacturing 
development 


Low  rate 
initial 
product 


Full 

rate 

product 


Sub- 

,  A 

Thread 

thread 

MRL  1 

MRL  2 

MRL  3 

MRL  4  i 

^  MRL  5 

MRL  6  V 

k  MRL  7 

MRL  8  A 

k  MRL  9 

MRL  10 

Technology 

maturity 

Should  be 

Should  be 

Should  be 

Should  be 

Should  be 

Should  be 

Should  be 

Should  be 

Should  be 

Should  be 

— 

assessed 

assessed 

assessed 

assessed 

assessed 

assessed 

assessed 

assessed 

assessed 

assessed 

atTRL  1 

atTRL  2 

atTRL  3 

atTRL  4 

atTRL  5 

atTRL  6 

atTRL  7 

atTRL  7 

atTRL  8 

atTRL  9 

Cost  and 

Cost  analy- 

Identify  any 

Cost  ele- 

Sensitiv- 

Producibility 

Costs 

Costs  analyzed 

Manufactur- 

Costs 

LRIP  cost 

FRP  cost 

funding- 

sis 

manufactur- 

ments  iden- 

ity  analysis 

cost  risks  as- 

analyzed  using 

using  prototype 

ing  costs 

analyzed 

goals  met 

goals 

ing  cost 

tified. 

conducted 

sessed.  Initial 

prototype 

system/subsys- 

rolled  up  to 

using  pilot 

and  learn- 

met.  Cost 

implications. 

to  define 

cost  models 

component  ac- 

tern  actuals  to 

system/sub- 

line  actuals 

ing  curve 

reduction 

cost  drivers 

support  Analy- 

tuals  to  ensure 

ensure  target 

system  level 

to  ensure 

analyzed 

initiatives 

and  produc- 

sis  of  Alterna- 

target  costs 

costs  are  achiev- 

and  tracked 

target 

with  actual 

onqoinq. 

tion  devel- 

tives  (AoA)  and 

are  achievable. 

able.  Allocate 

against  tar- 

costs  are 

data.  Cost 

opment 

Alternative 

Decisions  re- 

cost  targets  to 

gets.  Detailed 

achievable. 

reduction 

strategy  (i.e., 

Systems 

qardinq  desiqn 

subsystems. 

trade  studies 

Manufac- 

initiatives 

lab  to  pilot 

Review  (ASR). 

choices,  make/ 

Cost  reduction 

and  engineer- 

turing  cost 

onqoinq. 

to  factory). 

buy,  capacity, 

and  avoid- 

ing  change 

analysis 

Touch  labor 

process  capa- 

ance  strategies 

requests 

supports 

efficiency 

bil  ity,  sources, 

developed. 

supported 

proposed 

analyzed 

quality,  KCs, 

by  cost  esti- 

changes  to 

to  meet 

yield/rate,  and 

mates.  Cost 

require- 

production 

variabil- 

reduction  and 

ments  or 

rates  and 

ity  influenced 

avoidance 

configura- 

elements 

by  cost 

strategies 

tion.  Cost 

of  inef- 

models. 

underway. 

reduction 

ficiency  are 

initiatives 

identified 

ongoing. 

with  plans 
in  place  for 
reduction. 

Cost  and- 

Manu- 

Potential 

Program/ 

Program/ 

Manufacturing 

Program  has 

Program  has 

Program 

Program 

Program  has 

Production 

funding 

facturing 

investments 

projects 

projects 

technology 

updated  bud- 

reasonable 

has  updated 

has 

reasonable 

budgets 

investment 

identified. 

have 

have 

initiatives  iden- 

get  estimate 

budget  estimate 

budget 

reasonable 

budget 

sufficient  for 

budget 

reasonable 

reasonable 

tified  to  reduce 

for  reaching 

for  reaching 

estimate  for 

budget 

estimate 

production 

budget 

budget 

costs.  Program 

MRL  6  by  MS  B. 

MRL  8  by  MS 

reaching  MRL 

estimate 

for  FRP.  All 

at  required 

estimates 

estimates 

has  reasonable 

All  outstand- 

C.  Estimate 

8  by  MS  C.  All 

for  reach- 

outstanding 

rates  and 

for  reach- 

for  reaching 

budget  esti- 

ing  MRL  5 

includes  capital 

outstanding 

ing  MRL  9 

MRL  9  risk 

schedule 

ing  MRL  3 

MRL  4  by 

mate  for  reach- 

risk  areas 

investment  for 

MRL  7  risk 

by  the  FRP 

areas  under- 

to  support 

through 

MS  A. 

ing  MRL  6  by 

understood, 

production- 

areas  under- 

decision 

stood,  with 

funded 

experiment. 

MS  B.  Estimate 

with  approved 

representative 

stood,  with 

point. 

approved 

program. 

includes  capi- 

mitigation 

equipment  by 

approved 

Estimate 

mitigation 

tal  investment 

plans  in  place. 

CDR  and  pilot 

mitigation 

includes 

plans  in 

for  production- 

line  equipment 

plans  in  place. 

investment 

place. 

relevant 

by  MS  C.  All 

for  LRIP 

equipment. 

outstanding 

and  FRP.  All 

All  outstand- 

MRL  6  risk  areas 

outstand- 

ing  MRL  4 

understood, 

ing  MRL  8 

risk  areas 

with  approved 

risk  areas 

understood, 

mitigation  plans 

under- 

with  approved 

in  place. 

stood,  with 

mitigation 

approved 

plans  in  place. 

mitigation 
plans  in 
place. 
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Appendix  III:  Manufacturing  Readiness  Level 
(MRL)  Threads  and  Subthreads  (Risk  Areas) 


Acquisition 

Phase 


Pre-material 
solution  analysis 


Material 

solution 

analsysis 


Technology 

development 


Engineering  and 
manufacturing 
development 


Low  rate 
initial 
product 


Full 

rate 

product 


Thread 

Sub¬ 

thread 

MRL  1 

MRL  2 

MRL  3 

MRL  4  ^ 

^  MRL  5 

MRL  6  ^ 

^MRL7 

MRL  8^ 

^MRL9 

^  MRL  10 

- 

Technology 

maturity 

Should  be 
assessed 
atTRL  1 

Should  be 
assessed 
atTRL  2 

Should  be 
assessed 
atTRL  3 

Should  be 
assessed 
atTRL  4 

Should  be 
assessed 
atTRL  5 

Should  be 
assessed 
atTRL  6 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  8 

Should  be 
assessed 
atTRL  9 

Materi¬ 
als  (raw 
materials, 
compo¬ 
nents, 
sub-assem¬ 
blies  and 
subsys¬ 
tems) 

Maturity 

Material 
properties 
identified  for 
research. 

Material 
properties 
and  char¬ 
acteristics 
predicted. 
Material 
properties 
and  char¬ 
acteristics 
predicted. 

Material 
properties 
validated 
and  as¬ 
sessed 
for  basic 
manufactur¬ 
ability  using 
experi¬ 
ments. 

Survey  deter¬ 
mines  that 
the  projected 
material  has 
been  produced 
in  a  laboratory 
environment. 
Survey  deter¬ 
mines  that 
the  projected 
material  has 
been  produced 
in  a  laboratory 
environment. 

Materials 
have  been 
manufactured 
or  produced 
in  a  prototype 
environment 
(maybe  in  a 
similar  applica¬ 
tion/program). 
Maturity 
efforts  in  place 
to  address  new 
material  pro¬ 
duction  risks 
for  technology 
demonstration. 

Mate¬ 
rial  maturity 
verified  through 
technology 
demonstration 
articles.  Prelimi¬ 
nary  material 
specifications 
in  place  and 
material  proper¬ 
ties  have  been 
adequately 
characterized. 

Material 
maturity  suf¬ 
ficient  for 
pilot  line 
build.  Material 
specifications 
approved.Ma- 
terial  maturity 
sufficient  for 
pilot  line 
build.  Material 
specifications 
approved. 

Materials 
proven  and 
validated 
during 

EMD  as 
adequate 
to  sup¬ 
port  LRIP. 
Material 
specifica¬ 
tion  stable. 

Material  is 
proven  and 
controlled 
to  specifica¬ 
tion  in  LRIP. 
Material  is 
proven  and 
controlled 
to  specifica¬ 
tion  in  LRIP. 

Material  is 
proven  and 
controlled 
to  specifica¬ 
tion  in  FRP. 
Material  is 
proven  and 
controlled 
to  specifica¬ 
tion  in  FRP. 

Materi¬ 
als  (raw 
materials, 
compo¬ 
nents, 
sub-assem¬ 
blies  and 
subsys¬ 
tems) 

Availability 

Material 

availability 

assessed. 

Material 

scale-up 

issues 

identified. 

Projected  lead 
times  have 
been  identified 
for  all  difficult- 
to-obtain, 
difficult-to- 
process,  or 
hazardous 
materials. 
Quantities 
and  lead  times 
estimated. 

Availability  is¬ 
sues  addressed 
for  prototype 
build.  Signifi¬ 
cant  material 
risks  identified 
for  all  materi¬ 
als.  Planning 
has  begun  to 
address 
scale-up  issues. 

Availability 
issues  addressed 
to  meet  EMD 
build.  Long-lead 
items  identified. 
Potential  obso¬ 
lescence  issues 
identified. 

Availabil¬ 
ity  issues 
addressed 
to  meet 

EMD  builds. 
Long-lead 
procurement 
identified/ 
planned  for 
LRIP.  Obsoles¬ 
cence  plan  in 
place. 

Long-lead 
procure¬ 
ment 
initiated 
for  LRIP. 
Availability 
issues  pose 
no  signifi¬ 
cant  risk  for 
LRIP. 

Long-lead 
procure¬ 
ment  initi¬ 
ated  for  FRP. 
Availability 
issues  pose 
no  signifi¬ 
cant  risk  for 
FRP. 

Program 

is  in  FRP, 

with  no 

significant 

material 

availability 

issues. 

Materi¬ 
als  (raw 
materials, 
compo¬ 
nents, 
sub-assem¬ 
blies  and 
subsys¬ 
tems) 

Supply 
chain  man¬ 
agement 

Initial  as¬ 
sessment 
of  potential 
supply 
chain  capa¬ 
bility. 

Survey 
completed 
for  potential 
supply  chain 
sources. 

Potential 
supply  chain 
sources  identi¬ 
fied. 

Supply  chain 
plans  in  place 
(e.g.  teaming 
agreements  and 
so  forth)  leading 
to  an  EMD  con¬ 
tract  award. 

Effective 
supply  chain 
management 
process  in 
place.  Assess¬ 
ment  of  criti¬ 
cal  first  tier 
supply  chain 
completed. 

Supply 
chain  ade¬ 
quate  to 
support 

LRIP.  As¬ 
sessment 
of  critical 
second  and 
lower  tier 
supply  chain 
completed. 

Supply 
chain  is 
stable  and 
adequate  to 
support  FRP. 
Long-term 
agreements 
in  place 
where 
practical. 

Supply  chain 
proven  and 
supports 

FRP  require¬ 
ments. 

Materi¬ 
als  (raw 
materials, 
compo¬ 
nents, 
sub-assem¬ 
blies  and 

Special 
handling 
(i.e..  Gov¬ 
ernment 
Furnished 
Property 
(GFP),  shelf 

Initial 
evaluation 
of  potential 
regulatory 
require¬ 
ments  and 
special 

List  of 

hazardous 

materials 

identified. 

Special 

handling 

procedures 

List  of  hazard¬ 
ous  materials 
updated.  Spe¬ 
cial  handling 
procedures 
applied  in  the 
lab.  Special 

Special 
handling 
procedures 
applied  in 
production-rel¬ 
evant  environ¬ 
ment.  Special 

Special  handling 
procedures 
applied  in  pro¬ 
duction-relevant 
environment. 
Plans  to  address 
special  handling 

Special 
handling 
procedures 
applied  in 
production 
representa¬ 
tive  environ- 

Special 
handling 
procedures 
applied  in 
pilot  line 
environ¬ 
ment. 

Special 
handling 
procedures 
applied  in 

LRIP  envi¬ 
ronment. 
Special 

Special 
handling 
procedures 
effectively 
implement¬ 
ed  in  FRP. 
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Appendix  III:  Manufacturing  Readiness  Level 
(MRL)  Threads  and  Subthreads  (Risk  Areas) 


Acquisition 

Phase 


Pre-material 
solution  analysis 


Material 

solution 

analsysis 


Technology 

development 


Engineering  and 
manufacturing 
development 


Low  rate 
initial 
product 


Full 

rate 

product 


Thread 

Sub¬ 

thread 

MRL  1 

MRL  2 

MRL  3 

MRL  4  ^ 

^  MRL  5 

MRL  6  ^ 

^MRL7 

MRL  8^ 

^MRL9 

^  MRL  10 

- 

Technology 

maturity 

Should  be 
assessed 
atTRL  1 

Should  be 
assessed 
atTRL  2 

Should  be 
assessed 
atTRL  3 

Should  be 
assessed 
atTRL  4 

Should  be 
assessed 
atTRL  5 

Should  be 
assessed 
atTRL  6 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  8 

Should  be 
assessed 
atTRL  9 

subsys¬ 

tems) 

life,  secu¬ 
rity, 

hazardous 
materials, 
storage 
environ¬ 
ment,  and 
so  forth) 

handling 

concerns. 

applied 
in  the  lab. 
Special 
handling 
concerns 

assessed. 

handling 

requirements 

identified. 

handling 
requirement 
gaps  identified. 
New  special 
handling  pro¬ 
cesses  demon¬ 
strated  in  lab 
environment. 

requirement 
gaps  complete. 

ment.  Special 
handling 
procedures 
developed 
and  anno¬ 
tated  on  work 
instructions. 

Special 
handling 
procedures 
demon¬ 
strated  in 
EMDor 
technology 
insertion 
programs. 
Special  han¬ 
dling  issues 
pose  no 
significant 
risk  for  LRIP. 
All  work 
instructions 
contain 
special 
handling 
provisions, 
as  required. 

handling 
procedures 
demonstrat¬ 
ed  in  LRIP. 
Special  han¬ 
dling  issues 
pose  no 
significant 
risk  for  FRP. 

Process 
capabil¬ 
ity  amd 
control 

Model¬ 
ing  and 
simulation 
(product 
and  pro¬ 
cess) 

Initial 

models  de¬ 
veloped,  if 
applicable. 

Identifica¬ 
tion  of 
proposed 
manu¬ 
facturing 
concepts  or 
produc- 
ibility  needs 
based  on 
high-level 
process 
flowchart 
models. 

Production 
modeling  and 
simulation 
approaches 
for  process  or 
product  are 
identified. 

Initial  simula¬ 
tion  models 
(product  or 
process)  de¬ 
veloped  at  the 
component 
level. 

Initial  simulation 
models  devel¬ 
oped  at  the 
subsystem  or 
system  level. 

Simulation 
models  used 
to  determine 
system 
constraints 
and  identify 
improvement 
opportunities. 

Simulation 
models  veri¬ 
fied  by  pilot 
line  build. 
Results  used 
to  improve 
process  and 
determine 
that  LRIP 
require¬ 
ments  can 
be  met. 

Simulation 
model  veri¬ 
fied  by  LRIP 
build,  assists 
in  manage¬ 
ment  of 

LRIP  and 
determines 
that  FRP  re¬ 
quirements 
can  be  met. 

Simula¬ 
tion  model 
verified  by 

FRP  build. 
Production 
simulation 
models  used 
as  a  tool 
to  assist  in 
manage¬ 
ment  of  FRP. 

Process 
capabil¬ 
ity  amd 
control 

Manu¬ 

facturing 

process 

maturity 

Identifica¬ 
tion  of  ma¬ 
terial  and/ 
or  process 
approaches. 

Document 
high-level 
manufac¬ 
turing 
processes. 
Critical 
manufactur¬ 
ing  process¬ 
es  identified 
through 
experimen¬ 
tation. 

Complete 
a  survey  to 
determine 
the  current 
state  of  critical 
processes. 

Maturity  has 
been  assessed 
on  similar 
processes 
in  produc¬ 
tion.  Process 
capability 
requirements 
have  been 
identified  for 
pilot  line,  LRIP 
and  FRP. 

Manufacturing 
processes  dem¬ 
onstrated  in  pro¬ 
duction-relevant 
environment. 
Begin  collecting 
or  estimating 
process  capabil¬ 
ity  data  from 
prototype  build. 

Manufactur¬ 
ing  processes 
demon¬ 
strated  in  a 
production- 
representative 
environment. 
Continue 
collecting  or 
estimating 
process  capa¬ 
bility  data. 

Manu¬ 
facturing 
processes 
verified  for 
LRIP  on  a 
pilot  line. 
Process 
capability 
data  from 
pilot  line 
meets 
target. 

Manufactur¬ 
ing  processes 
are  stable, 
adequately 
controlled, 
and  capable 
and  have 
achieved 
program  LRIP 
objectives. 
Variability 
experiments 
conducted 

Manu¬ 
facturing 
processes 
are  stable, 
adequately 
controlled, 
capable, 
and  have 
achieved 
program 
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Acquisition 

Phase 


Pre-material 
solution  analysis 


Material 

solution 

analsysis 


Technology 

development 


Engineering  and 
manufacturing 
development 


Low  rate 
initial 
product 


Full 

rate 

product 


Thread 

Sub¬ 

thread 

MRL  1 

MRL  2 

MRL  3 

MRL  4  ^ 

^  MRL  5 

MRL  6  ^ 

^MRL7 

MRL  8^ 

^MRL9 

^  MRL  10 

- 

Technology 

maturity 

Should  be 
assessed 
atTRL  1 

Should  be 
assessed 
atTRL  2 

Should  be 
assessed 
atTRL  3 

Should  be 
assessed 
atTRL  4 

Should  be 
assessed 
atTRL  5 

Should  be 
assessed 
atTRL  6 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  8 

Should  be 
assessed 
atTRL  9 

to  show  FRP 
impact  and 
potential  for 
continuous 
improve¬ 
ment. 

FRP  objec¬ 
tives. 

Process 
capabil¬ 
ity  and 
control 

Process 
yields  and 
rates 

Initial 

estimates  of 
yields  and 
rates  based 
on  experi¬ 
ments  or 
state  of  the 
art. 

Yield  and  rates 
assessment  on 
proposed/simi¬ 
lar  processes 
complete  and 
applied  within 
AoA. 

Target  yields 
and  rates 
established  for 
pilot  line,  LRIP, 
and  FRP.  Yield 
and  rate  issues 
identified. 
Improvement 
plans  devel¬ 
oped/initiated. 

Yields  and 
rates  from 
production-rele¬ 
vant  environ¬ 
ment  evaluated 
against  targets 
and  the  results 
feed  improve¬ 
ment  plan. 

Yields  and 
rates  from 
production- 
representative 
environment 
evaluated 
against  pilot 
line  targets 
and  the 
results  feed 
improvement 
plans. 

Pilot  line 
targets 
achieved. 
Yields 
and  rates 
required  to 
begin  LRIP 
verified 
using  pilot 
line  articles. 
Improve¬ 
ment  plans 
ongo¬ 
ing  and 
updated. 

LRIP  yield 
and  rate 
targets 
achieved. 

Yield  im¬ 
provements 
ongoing. 

FRP  yield 
and  rate 
targets 
achieved. 

Yield  im¬ 
provements 
ongoing. 

Quality 

manage¬ 

ment 

Quality 

manage¬ 

ment, 

including 

supplier 

quality 

Quality  strate¬ 
gy  identified  as 
part  of  theTDS 
and  included 
in  SEP. 

Quality  strat¬ 
egy  updated 
to  reflect  KC 
identification 
activities. 

Initial  quality 
plan  and  quality 
management 
system  is  in 
place.  Quality 
risks  and  metrics 
have  been 
identified. 

Qual¬ 
ity  targets 
established. 
Demonstrate 
ability  to 
collect  and 
analyze 
quality  data 
(process  and 
system)  in  the 
production- 
representative 
environment. 

Qual¬ 
ity  targets 
demon¬ 
strated  on 
pilot  line. 
Continuous 
quality  im¬ 
provement 
ongoing. 
Supplier 
products 
have  com¬ 
pleted 
qualifi¬ 
cation 
testing  and 
first-article 
inspection. 
Supplier 
products 
pass  accep¬ 
tance  test¬ 
ing  at  a  rate 
adequate  to 
begin  LRIP. 

Quality  tar¬ 
gets  verified 
on  LRIP  line. 
Continu¬ 
ous  quality 
improvement 
ongoing. 
Supplier 
products 
pass  accep¬ 
tance  testing 
at  a  rate 
adequate  to 
transition  to 
FRP. 

Quality  tar¬ 
gets  verified 
on  FRP  line. 
Continuous 
quality  im¬ 
provement 
ongoing. 

Page  61 


GAO-10-439  DOD  Manufacturing  Readiness 


Appendix  III:  Manufacturing  Readiness  Level 
(MRL)  Threads  and  Subthreads  (Risk  Areas) 


Acquisition 

Phase 


Pre-material 
solution  analysis 


Material 

solution 

analsysis 


Technology 

development 


Engineering  and 
manufacturing 
development 


Low  rate 
initial 
product 


Full 

rate 

product 


Thread 

Sub¬ 

thread 

MRL  1 

MRL  2 

MRL  3 

MRL  4  ^ 

^  MRL  5 

MRL  6  ^ 

^MRL7 

MRL  8^ 

^MRL9 

^  MRL  10 

- 

Technology 

maturity 

Should  be 
assessed 
atTRL  1 

Should  be 
assessed 
atTRL  2 

Should  be 
assessed 
atTRL  3 

Should  be 
assessed 
atTRL  4 

Should  be 
assessed 
atTRL  5 

Should  be 
assessed 
atTRL  6 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  8 

Should  be 
assessed 
atTRL  9 

Manu¬ 

facturing 

personnel 

Manu¬ 

facturing 

personnel 

New 

manufactur¬ 
ing  skills 
identified. 

Manufactur¬ 
ing  skill  sets 
identified  and 
production 
workforce 
requirements 
(technical  and 
operational) 
evaluated  as 
part  of  AoA. 
Determine 
availabil¬ 
ity  of  process 
development 
workforce  for 
theTDP. 

Skill  sets 
identified  and 
plans  devel¬ 
oped  to  meet 
prototype 
and  produc¬ 
tion  needs. 
Special  skills 
certification 
and  training 
requirements 
established. 

Manufacturing 
workforce  skills 
available  for 
production  in 
a  relevant  envi¬ 
ronment.  Iden¬ 
tify  resources 
(quantities  and 
skill  sets)  and 
develop  initial 
plans  to  achieve 
requirements 
for  pilot  line  and 
production. 

Manufactur¬ 
ing  workforce 
resource 
requirements 
identified 
for  pilot 
line.  Plans 
developed  to 
achieve  pilot 
line  require¬ 
ments.  Plans 
updated  to 
achieve  LRIP 
workforce 
requirements. 
Pilot  line 
workforce 
trained  on 
representative 
environment. 

Manu¬ 

facturing 

workforce 

resource  re¬ 
quirements 
identified 
for  LRIP. 
Plans 

developed 
to  achieve 
LRIP  re¬ 
quirements. 
Plans 

updated  to 
achieve  FRP 
workforce 
require¬ 
ments.  LRIP 
personnel 
trained  on 
pilot  line 
where  pos¬ 
sible. 

LRIP 

personnel 
requirements 
met.  Imple¬ 
ment  plan  to 
achieve  FRP 
workforce  re¬ 
quirements. 

FRP  person¬ 
nel  require¬ 
ments  met. 
Production 
workforce 
skill  sets 
maintained 
due  to 
attrition  of 
workforce. 

Facilities 

Tooling/ 

STE/SIE 

Tooling/STE/ 

SIE  require¬ 
ments  are 
considered  as 
part  of  AoA. 

Identify  tool¬ 
ing  and  Special 
Test  Equip¬ 
ment/  Special 
Inspection 
Equipment 
(STE/SIE) 
requirements 
and  provide 
supporting 
rationale  and 
schedule. 

Prototype  tool¬ 
ing  and  STE/SIE 
concepts  dem¬ 
onstrated  in  pro¬ 
duction  relevant 
environment. 
Production  tool¬ 
ing  and  STE/SIE 
requirements 
developed. 

Production 
tooling  and 
STE/SIE 
design  and 
development 
efforts  under¬ 
way.  Manu¬ 
facturing 
equipment 
maintenance 
strategy 
developed. 

All  tooling, 
test,  and 
inspection 
equipment 
proven  on 
pilot  line 
and  re¬ 
quirements 
identified 
for  LRIP. 
Manu¬ 
facturing 
equipment 
mainte¬ 
nance 

demon¬ 
strated  on 
pilot  line. 

All  tooling, 
test,  and 
inspection 
equipment 
proven  in 

LRIP  and 
requirements 
identified  for 
FRP. 

Manu¬ 

facturing 

equipment 

maintenance 

schedule 

demon¬ 

strated. 

Proven  tool¬ 
ing,  test,  and 
inspection 
equipment 
in  place  to 
support 
maximum 

FRP.  Planned 
equipment 
mainte¬ 
nance 

schedule 

achieved. 

Facilities 

Facilities 

Specialized 
facility  re¬ 
quirements/ 
needs 
identified. 

Availability  of 
manufacturing 
facilities  for 
prototype  de¬ 
velopment  and 
production 

Manufacturing 
facilities  identi¬ 
fied  and  plans 
developed 
to  produce 
prototypes. 

Manufacturing 
facilities  identi¬ 
fied  and  plans 
developed  to 
produce  pilot 
line  build. 

Manufactur¬ 
ing  facilities 
identified 
and  plans 
developed  to 

Pilot  line 
facilities 
demon¬ 
strated. 
Manu¬ 
facturing 

Manufactur¬ 
ing  facilities 
in  place  and 
demon¬ 
strated  in 

Production 
facilities  in 
place  and 
capacity 
demonstrat¬ 
ed  to  meet 
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Acquisition 

Phase 


Pre-material 
solution  analysis 


Material 

solution 

analsysis 


Technology 

development 


Engineering  and 
manufacturing 
development 


Low  rate 
initial 
product 


Full 

rate 

product 


Thread 

Sub¬ 

thread 

MRL  1 

MRL  2 

MRL  3 

MRL  4  ^ 

^  MRL  5 

MRL  6  ^ 

^  MRL  7 

MRL  8^ 

^MRL9 

9  MRL  10 

- 

Technology 

maturity 

Should  be 
assessed 
atTRL  1 

Should  be 
assessed 
atTRL  2 

Should  be 
assessed 
atTRL  3 

Should  be 
assessed 
atTRL  4 

Should  be 
assessed 
atTRL  5 

Should  be 
assessed 
atTRL  6 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  7 

Should  be 
assessed 
atTRL  8 

Should  be 
assessed 
atTRL  9 

evaluated  as 
part  of  AoA. 

produce  LRIP 
build. 

facilities 
adequate 
to  begin 

LRIP.  Plans 
in  place  to 
support 
transition 
to  FRP. 

LRIP.  Capac¬ 
ity  plans 
adequate  to 
support  FRP. 

maximum 

FRP  require¬ 
ments. 

Manu¬ 

facturing 

manage¬ 

ment 

Manu¬ 
facturing 
planning 
and  sched¬ 
uling 

Manufactur¬ 
ing  strategy 
developed  and 
integrated 
with  acquisi¬ 
tion  strat¬ 
egy.  Prototype 
schedule  risk 
mitigation 
efforts  incor¬ 
porated  into 
TDS. 

Manufactur¬ 
ing  strategy 
refined  based 
upon  preferred 
concept.  Proto¬ 
type  schedule 
risk  mitigation 
efforts  initi¬ 
ated. 

Initial  manufac¬ 
turing  approach 
developed. 

All  system- 
design-related 
manufacturing 
events  included 
in  Integrated 
Master  Plan/ 
Integrated 

Master  Schedule 
(IMP/IMS). 
Manufacturing 
risk  mitigation 
approach  for 
pilot  line  or 
technology 
insertion  pro¬ 
grams  defined. 

Initial  manu¬ 
facturing  plan 
developed. 
Manufactur¬ 
ing  planning 
required  to 
achieve  MRL 

8  has  been 
included  in 
the  IMP/IMS. 
Manufactur¬ 
ing  risks 
integrated 
into  risk  miti¬ 
gation  plans. 
Develop 
initial  work 
instructions. 
Effective 
production 
control  sys¬ 
tem  in  place 
to  support 
pilot  line. 

Manufac¬ 
turing  plan 
updated  for 
LRIP.  All  key 
manufac¬ 
turing  risks 
are  identi¬ 
fied  and  as¬ 
sessed  with 
approved 
mitigation 
plans  in 
place.  Work 
instructions 
finalized. 
Effective 
production 
control 
system  in 
place  to 
support 

LRIP. 

Manufactur¬ 
ing  plan 
updated 
for  FRP.  All 
manufactur¬ 
ing  risks 
tracked  and 
mitigated. 
Effective 
production 
control  sys¬ 
tem  in  place 
to  support 

FRP. 

All  manufac¬ 
turing  risks 
mitigated. 

Materials 

planning 

Technology 
development 
article 
component 
list  developed 
with  associ¬ 
ated  lead-time 
estimates. 

Technology 
development 
part  list  matur¬ 
ing.  Make/ 
buy  evalua¬ 
tions  begin 
and  include 
production 
considerations 
reflecting  pilot 
line,  LRIP,  and 

FRP  needs. 

Lead  times 
and  other  risks 
identified. 

Most  material 
decisions  com¬ 
plete  (make/ 
buy),  material 
risks  identified, 
and  mitigation 
plans  devel¬ 
oped.  Bill  of 
Materials  (BOM) 
initiated. 

Make/buy 
decisions 
and  BOM 
complete 
for  pilot  line 
build.  Mate¬ 
rial  planning 
systems  in 
place  for  pilot 
line  build. 

Make/buy 
decisions 
and  BOM 
complete 
to  sup¬ 
port  LRIP. 
Material 
planning 
systems  in 
place  for 
LRIP  build. 

Make/buy 
decisions 
and  BOM 
complete 
to  support 

FRP.  Material 
planning  sys¬ 
tems  in  place 
for  FRP. 

Mate¬ 
rial  planning 
systems 
validated  on 
FRP  build. 

Source:  GAO. 
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OFFICE  OF  THE  DIRECTOR  OF 
DEFENSE  RESEARCH  AND  ENGINEERING 

3040  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3040 


APR  19 


Mr.  Michael  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  NW 
Washington,  DC  20548 

Dear  Mr.  Sullivan: 

This  is  the  Department  of  Defense  response  to  the  GAO  draft  report,  GAO- 10- 
439,  ‘BEST  PRACTICES:  DoD  Can  Achieve  Better  Outcomes  by  Standardizing  the  Way 
Manufacturing  Risks  Are  Managed'  dated  March  12,  2010  (GAO  Code  120793). 

Detailed  comments  on  the  report  recommendations  are  enclosed. 

The  Department  appreciates  the  opportunity  to  respond  to  your  draft  report  and 
looks  forward  to  working  with  you  as  we  continue  to  improve  manufacturing  readiness  of 
our  acquisition  programs. 


Stephen  P.  Welby 
Director 

Systems  Engineering 


Enclosure: 
As  stated 
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DoD  Response  to  GAO- 10-439  Recommendations 

GAO  DRAFT  REPORT  DATED  MARCH  12,  2010 
GAO-10-439  (GAO  CODE  120793) 

BEST  PRACTICE:  DOD  CAN  ACHIEVE  BETTER  OUTCOMES  BY 
STANDARDIZING  THE  WAY  MANUFACTURING  RISKS  ARE  MANAGED 

DEPARTMENT  OF  DEFENSE  COMMENTS  TO  THE  GAO 
RECOMMENDATIONS 

RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense  require 
the  assessment  of  manufacturing  readiness  across  DoD  programs  using  consistent  MRL 
criteria  as  a  basis  for  measuring,  assessing,  reporting,  and  communicating  manufacturing 
readiness  and  risk  on  science  and  technology  transition  projects  and  acquisition 
programs. 

DOD  RESPONSE:  Partially  concur:  The  Department  of  Defense  recognizes  that  mature 
manufacturing  processes  and  readiness  are  critical  to  achieving  predictable  and  successful 
program  outcomes.  It  also  recognizes  the  value  in  assessing  manufacturing  risks  during 
science  and  technology  research  on  technologies  planned  to  be  incorporated  into 
acquisition  programs.  Department  of  Defense  Instruction  (DoDI)  5000.02,  Operation  of 
the  Defense  Acquisition  System,  dated  8  December  2008  reflects  an  increased  focus  on 
manufacturing  throughout  the  acquisition  lifecycle  for  programs  of  all  acquisition 
categories.  Specifically,  it  establishes  a  framework  to  continually  assess  and  mitigate 
manufacturing  risks  during  the  Analysis  of  Alternatives,  2366b  certifications  to  Congress, 
Preliminary  and  Critical  Design  Reviews;  and  acquisition  milestones. 

The  Department’s  new  manufacturing  readiness  criteria  will  form  the  basis  for  assessing 
pertinent  science  and  technology  efforts,  and  acquisition  programs  throughout  the 
acquisition  lifecycle  on  programs  of  all  acquisition  categories.  These  criteria  will  be  a 
tool  to  identify  relevant  manufacturing  risks  which  require  mitigation.  These 
manufacturing  readiness  criteria  are  expected  to  be  tailored  for  programs  and  will  be 
included  in  the  Department’s  criteria  for  systems  engineering  technical  reviews;  the 
Department’s  templates  for  Preliminary  Design  Review/Critical  Design  Review  reports; 
and  acquisition  phase  exit  criteria.  These  manufacturing  readiness  criteria  will  also  be 
assessed  as  part  of  the  Program  Support  Reviews  which  the  Department  conducts  on 
Major  Defense  Acquisition  Programs.  These  reviews  evaluate  manufacturing  as  part  of 
an  overall  integrated  program  assessment.  These  manufacturing  readiness  criteria  and 
products  will  be  made  available  to  government  and  industry.  Their  use  by  the  Services 
on  lower  ACAT  programs  will  also  be  encouraged.  The  Navy’s  Gate  Review  process 
currently  assesses  manufacturing  risks  but  is  being  updated  with  the  new  manufacturing 
readiness  criteria. 
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DoD  Response  to  GAO- 10-439  Recommendations 

RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Office  of  the  Director,  Defense  Research  and  Engineering  to  examine  strengthening 
the  MRL  criteria  related  to  the  process  capability  and  control  of  critical  components 
and/or  interfaces  prior  to  the  Milestone  C  low  rate  initial  production  decision. 

DOD  RESPONSE:  Concur.  Department  of  Defense  Instruction  5000.02  directs  that 
programs  at  Milestone  C  have  no  significant  manufacturing  risks;  that  manufacturing 
processes  have  been  effectively  demonstrated  in  a  pilot  line  environment;  and 
manufacturing  processes  are  under  control  (if  Milestone  C  is  full-rate  production).  While 
the  Department  notes  that  all  manufacturing  processes  do  not  warrant  the  same  level  of 
process  capability  and  control,  appropriate  levels  of  control  are  certainly  warranted  on  a 
case  by  case  basis. 

The  Department  will  examine  strengthening  the  manufacturing  readiness  criteria  related 
to  process  capability  and  control  of  critical  components  and/or  interfaces  prior  to  the 
Milestone  C  low  rate  initial  production  decision.  However,  program  offices  and 
contractors  should  continue  to  have  the  latitude  to  jointly  agree  on  the  targets  and  specific 
process  control  demonstrations  required  on  the  pilot  production  line  during  the 
Engineering  and  Manufacturing  Development  to  ensure  success. 

RECOMMENDATION  3:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Office  of  the  Director,  Defense  Research  and  Engineering  to  assess  the  need  for 
analytical  models  and  tools  to  support  MRL  assessments. 

DOD  RESPONSE:  Concur.  The  Department  will  collaborate  with  government  services, 
contractors,  and  academia  to  capture  knowledge  and  provide  improved  tools  for 
government  and  contractor  usage  in  conducting  assessments  of  manufacturing  readiness 
as  part  of  systems  engineering  technical  reviews  and  milestone  reviews. 

RECOMMENDATION  4:  The  GAO  recommends  that  the  Secretary  of  Defense  assess 
the  adequacy  of  the  manufacturing  workforce  knowledge  and  skills  base  across  the 
military  services  and  defense  agencies  and  develop  a  plan  to  address  current  and  future 
workforce  gaps 

DOD  RESPONSE:  Concur.  We  agree  that  the  production,  quality  and  manufacturing 
(PQM)  career  field  has  suffered  erosion,  as  have  other  DoD  career  fields.  The  USD 
(AT&L)  Director  of  Human  Capital  has  launched  a  review  of  the  PQM  career  field 
design  to  identify  the  skills,  knowledge  and  training  required  at  each  level  of  career 
progression  in  order  to  develop  training  courses  and  evaluate  progression  of  anticipated 
DoD  planned  new  hires.  The  Department  has  started  to  implement  hiring  and  retention 
strategies  to  mitigate  the  potential  loss  in  experienced,  senior-level  PQM  talent  and 
increase  the  size  of  the  manufacturing  workforce.  As  part  of  the  Secretary's  growth 
strategy  and  other  initiatives,  the  PQM  career  field  is  projected  to  grow  approximately 
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DoD  Response  to  GAO- 10-439  Recommendations 

1,300  (13%)  by  FY2015.  Each  of  the  military  services  and  other  DOD  components  has 
been  actively  planning  and  deploying  initiatives  that  support  the  DOD  acquisition 
workforce  growth  strategy.  Components  have  submitted  planning  inputs  to  OSD  and  to 
the  Defense  Acquisition  Workforce  Senior  Steering  Board,  and  growth  is  underway. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  Web  site  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 

MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 


Chuck  Young,  Managing  Director,  youngcl@gao.gov,  (202)  512-4800 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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